Filesystem Watcher Error - Debian Buster

[fr58426]

Hi,

I’m running Syncthing 1.11.1 on a Debian Buster based server, syncing ~2 TB worth of data in some 3M directories, spread over a few folders. Recently, Filesystem Watcher Errors (“failed to setup inotify handler. Please increase inotify limits, see https://docs.syncthing.net/users/faq.html#inotify-limits”) have started showing up. I have ramped up fs.inotify.max_user_watches like suggested and beyond, to no avail. As far as I understand, a simple test for an actual limititation in inotify watchers is running tail -f on a file, which succeeds in my case. Following up I switched to the syncthing user and did this:

syncthing@server:~$ touch a
syncthing@server:~$ inotifywatch a
Establishing watches...
Finished establishing watches, now collecting statistics.
^Z
[1]+  Stopped                 inotifywatch a
syncthing@server:~$ bg 1
[1]+ inotifywatch a &
syncthing@server:~$ echo "asd" > a
syncthing@server:~$ cat a
asd
syncthing@server:~$ fg 1
inotifywatch a
^Ctotal  access  modify  close_write  close_nowrite  open  filename
7      1       2       1            1              2     a

The way I see it, this shows that there shouldn’t be any issue setting up inotify handlers. Am I reading this wrong? What could be going wrong instead?

[calmh (Jakob Borg)]

I’m not sure what the tail test is supposed to prove or disprove. Does it set up watches on all 3M+ items in your folders? This is what Syncthing attempts.

[fr58426]

Thanks for taking the time to reply. I was just trying to figure out sort of independently whether the problem really is the inotify watcher limit. I described two things that searching online produced as ways to figure out whether there are stiull watches available.

Do I understand correctly that Syncthing tries to set up all watches at the same time and instead of taking whatever it can get, produces the error I see? Would you recommend further increasing max_user_watches? I looked at my numbers again, it’s actually 3M+ files in 900K+ folders, max_user_watches is currently set to 524288 because that’s a number I found here as a reasonable upper bound. Does Syncthing need 1 watch per file? Per folder?

Thanks again for helping me understand all of this better.

[calmh (Jakob Borg)]

Minimum is one per watched folder, yes. Syncthing will attempt to watch all folders. If it fails, none are watched and the resources are released. The inotifywatch check verifies that there is at least one watch available, but you need 900k+ watches then.