Feature? Elasticsearch Integration

DAlba-sudo · December 1, 2023, 6:42pm

I’m looking to do some open source contributions - still early in my career, but I’m hungry to contribute.

I’ve had to chance to use syncthing and have been reading the forums for a bit. I recall reading a post about re-doing the logging, which got me thinking, would the community appreciate some form of elasticsearch integration (either natively, or as a different program)? Especially for those of you that use this in a IT/Enterprise environment?

I’m also open to contributing on other things. Good project you’ve all got here! Cheers.

AudriusButkevicius · December 2, 2023, 3:32pm

I’m not sure how this would be useful? Can’t you just feed your logs to elastic? Why does it need integration, what would the integration even do?

DAlba-sudo · December 2, 2023, 5:10pm

I’ve recently been making custom “agents” that take the logs and feed it to an elasticsearch endpoint for other platforms/services (i.e., firewalls, etc) in a format that is more conducive to making dashboards or whatnot. What about some pre-made dashboards? Just throwing out ideas here.

I guess I’m just trying to test the waters if you will, I don’t have concrete ideas/plans yet.

mraneri · December 3, 2023, 6:33am

I think much simpler than this the log interface in the GUI can use a simple text search. Show me all log lines that match GX6GIWE (a device) or a file name, or an error…

DAlba-sudo · December 3, 2023, 5:04pm

On it!

DAlba-sudo · December 3, 2023, 7:20pm

github.com/syncthing/syncthing

Feature: Logging Search Bar

opened 07:18PM - 03 Dec 23 UTC

DAlba-sudo

enhancement needs-triage

# Description Inspired by a short exchange on the forum: https://forum.syncth…ing.net/t/feature-elasticsearch-integration/21186/4?u=dalba-sudo. I'm thinking of adding a search bar in the log section of the GUI. # Use Case "I think much simpler than this the log interface in the GUI can use a simple text search. Show me all log lines that match GX6GIWE (a device) or a file name, or an error…" - [mraneri](https://forum.syncthing.net/u/mraneri) Here's my current idea for the filtering implementation: * add a filter function to the logging scope/class (kind of new angular, not sure what you'd call it). * called by ng-keyup callback on an input of type text (i.e., the "search bar"). * only filters on keywords more than 2 characters long (if you want to look for something that is 2 characters long, use a wildcard as the third character?) * add an input of type text in the `logViewerModalView` I've linked to a video demoing the current implementation - it's bare-bones and not currently designed for scale. Before I go back to the drawing board to think about ways to make this scalable for larger logs, I want to get feedback on the feature! Demo: https://youtu.be/64B5Fa2fiq4 > This is my first contribution to open-source, any feedback is appreciated (i.e., more detail explaining the feature, etc). Cheers, DAlba-Sudo

^ I raised an issue to get some more feedback on the feature. Feel free to look at that for updates as I continue working on it.

For reference, I’ve gone ahead and implemented a basic version of this: https://youtu.be/64B5Fa2fiq4.

Let me know what you think.

mraneri · December 4, 2023, 1:36am

That’s what I had in mind…. As long as it’s love updating and searches the whole log history (not just what’s in the window. It’s hard to tell from the video but yeah. This would be a big help trying to diagnose intermittent connection issues with specific machines.

Thoughts from others welcome.

DAlba-sudo · December 4, 2023, 2:11am

my understanding is definitely that it’s only taking the entries currently in the window (and more importantly the ones fetched while syncthing’s gui is running). I’ll take a look tomorrow morning at what needs to happen to get the global log and do the same, although I suspect that with the current architecture such a change would either require reading the file(s) from go, and exposing it via the API (haven’t taken a look at either the API, or how files are currently saved to disk) or some other magic.

As an aside, I did notice the calls made to the log endpoint request logs after a certain time-frame which could be modified in the logging menu to a prior point in time. Whether the API will go back and into the file, I can’t answer that just yet.

Definitely an interesting challenge and I am enjoying looking through the source code. It seems most maintainers/users use the cli for the logging stuff, is this because of preference reasons? Or perhaps is there something else.