I’m running Syncthing 1.11.1 on a Debian Buster based server, syncing ~2 TB worth of data in some 3M directories, spread over a few folders. Recently, Filesystem Watcher Errors (“failed to setup inotify handler. Please increase inotify limits, see https://docs.syncthing.net/users/faq.html#inotify-limits”) have started showing up. I have ramped up
fs.inotify.max_user_watches like suggested and beyond, to no avail. As far as I understand, a simple test for an actual limititation in inotify watchers is running
tail -f on a file, which succeeds in my case. Following up I switched to the syncthing user and did this:
syncthing@server:~$ touch a
syncthing@server:~$ inotifywatch a
Finished establishing watches, now collecting statistics.
+ Stopped inotifywatch a
syncthing@server:~$ bg 1
+ inotifywatch a &
syncthing@server:~$ echo "asd" > a
syncthing@server:~$ cat a
syncthing@server:~$ fg 1
^Ctotal access modify close_write close_nowrite open filename
7 1 2 1 1 2 a
The way I see it, this shows that there shouldn’t be any issue setting up inotify handlers. Am I reading this wrong? What could be going wrong instead?
I’m not sure what the tail test is supposed to prove or disprove. Does it set up watches on all 3M+ items in your folders? This is what Syncthing attempts.
Thanks for taking the time to reply. I was just trying to figure out sort of independently whether the problem really is the inotify watcher limit. I described two things that searching online produced as ways to figure out whether there are stiull watches available.
Do I understand correctly that Syncthing tries to set up all watches at the same time and instead of taking whatever it can get, produces the error I see? Would you recommend further increasing
I looked at my numbers again, it’s actually 3M+ files in 900K+ folders,
max_user_watches is currently set to
524288 because that’s a number I found here as a reasonable upper bound. Does Syncthing need 1 watch per file? Per folder?
Thanks again for helping me understand all of this better.
Minimum is one per watched folder, yes. Syncthing will attempt to watch all folders. If it fails, none are watched and the resources are released. The inotifywatch check verifies that there is at least one watch available, but you need 900k+ watches then.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.