Event/streaming interface instead of polling

Feature
1

Use case

In Heartbeat there are both a large number of repositories and some of them should be as performant as possible while passing small amounts of data (think: a rudimentary form of chat). We are going to hit the limits of the current polling interface pretty soon.

Currently the GUI warns that the lowest possible rescan interval is 5 seconds. I can see how it might be impractical to go below this with polling, especially if there are a lot of repositories. Is the long term answer to build an event/watch-based system? Regardless, we need to make it so that updates can be synced as quickly as possible.

Thoughts?

CC @calmh @AudriusButkevicius

2

We’ve explored this avenue before and there is no “proper” way of doing it. Existing watch-based system have limits (I think its 8k files for inotify?).

There are API’s for triggering a rescan in a particular directory/folder though, which is what https://github.com/Zillode/syncthing-inotify uses, which I guess you can use too.

Also there is https://github.com/go-fsnotify/fsnotify , but it lacks certain platform support and will only be merged in Go 1.5, and still more or less suffers from the same issues such as limits.

Also see: https://github.com/syncthing/syncthing/issues/9 for the longer discussion.

1 Like
3

This might be good for your use case once fully implemented, because files aren’t even stored to disk directly.

This means files can also be accessed before being synced (by pulling from another node), and nodes don’t have to store all files locally.