Yes, the GUI and activation by default is yet to be released (in RC). The documentation is in place: https://docs.syncthing.net/users/syncing.html#scanning and https://docs.syncthing.net/users/config.html#folder-element
Hello:
Thanks for the work on the FS watcher - really appreciate this inclusion.
However, I’m having an issue with the watcher failing to start up on large folders.
I’m running this on a Synology NAS, and have consequently raised the inotify max_user_watches as recommended (it defaults to 8192). So far I have this raised to 2097152 (confirmed by running cat /proc/sys/fs/inotify/max_user_watches ) - but I’m still running into failures on a couple of folders.
It does appear to be working on a folder with 25288 files, 646GB in size. It is not working on a folder of 131588 files, 238GB in size. Or one of 3908571 files, 21939GB in size.
Any ideas if this is to be expected for the quantity of files being watched, or if there are Bad Things to be expected for raising the limit even further? I tried to find out the limits of inotify elsewhere - but it doesn’t seem to be at all straightforward!
Many thanks,
Pants.
Hello:
Still having issues with the FS watcher not starting up on large folders.
I tried doubling the limit even further (to ~4 million) - but that seemed to kill the watcher on even small folders.
I’ve reduced it back to the recommended value (1048576) - and the watcher has come back for my smaller folders. However, I’m still seeing Failed to setup, retrying on the larger folders.
I’m monitoring the logs in the watchaggregator category - but there doesn’t seem to be anything coming through in there - and I can’t see any other debug categories that might show problems in this area.
I’ve also been digging around for information on inotify - and on this page, it recommends running tail -f /var/log/dmesg to see if you’ve run out of fs watchers. I’ve done this - and it looks like I haven’t run out.
Unfortunately, as I’m running this on a NAS, I don’t have a full compliment of OS commands to try.
Any ideas where to look at next?
Thanks,
Pants.
The max_user_watches setting is definitely not the issue, neither is the size alone problematic (works fine here).
So far you haven’t posted the actual error message that is displayed as a notice/warning on the top, as a tooltip on the watcher status and in the logs - that’s what should explain the cause.
Hi Simon:
Thanks for chiming in.
I can’t find anything in logs so far (any idea which debug log I should enable?) - but the tooltip on the watcher status says:
Periodic scanning at given interval and failed setting up watching for changes, retrying every 1m: not started
Thanks,
Pants.
So it hasn’t even tried to setup the watcher yet, i.e. the folder code is busy doing other stuff (initial scan?). UI screenshot with the affected folder expanded and/or logs could help shed some light.
Hi Simon:
Ah - I’ve just seen that a couple of the folders are specifically referencing the inotify limits:
Periodic scanning at given interval and failed setting up watching for changes, retrying every 1m: failed to setup inotify handler. Please increase inotify limits, see blah blah blah
Sorry - I hadn’t noticed that those messages were specific to the inotify limit - I shall investigate further and report back!
Thanks,
Pants.
So it hasn’t even tried to setup the watcher yet, i.e. the folder code is busy doing other stuff (initial scan?).
Here is a quick example on my machine:
Folder ID f67cp-nrcdd
Folder Path /media/syncthing/mik
Global State 1 0 ~22.84 KiB
Local State 1 0 ~22.84 KiB
Rescans 1h Failed to setup, retrying
Shared With Miks-machine.local
Last Scan 2018-07-09 11:58:17
Not a pretty GUI dump, but you get the idea. A single file in the folder, yet the fswatcher isn’t running. I’ve turned on the debug option watchaggregator and I’m seeing events on other folders, but nothing about that folder.
Then, magically, it started working, but the ‘online’ logs don’t go back far enough for me to see what happened, and nothing has appeared in the syslog so I can’t pull it from there either. I’d guess over an hour had passed before it managed to start.
Sadly this doesn’t help in anyway to understand the issue. Syncthing logs to stdout, you should check if that is preserved somewhere (systemd et all).
I’ve got everything thrown into rsyslog, but there isn’t anything useful in there.
e.g.
$ grep f67cp-nrcdd *syncthing
Jul 9 11:35:49 syncthing syncthing[4952]: [ER2BS] INFO: Ready to synchronize "mik-films" (f67cp-nrcdd) (readwrite)
Jul 9 11:35:50 syncthing syncthing[4952]: [ER2BS] INFO: Completed initial scan of readwrite folder "mik-films" (f67cp-nrcdd)
Jul 9 11:51:40 syncthing syncthing[4952]: [ER2BS] 2018/07/09 11:51:40.094779 aggregator.go:158: DEBUG: aggregator/"mik-films" (f67cp-nrcdd): Stopped
Jul 9 11:51:43 syncthing syncthing[4994]: [ER2BS] INFO: Ready to synchronize "mik-films" (f67cp-nrcdd) (readwrite)
Jul 9 11:51:43 syncthing syncthing[4994]: [ER2BS] INFO: Completed initial scan of readwrite folder "mik-films" (f67cp-nrcdd)
$
Similarly, if I grep for mik-films, same list of entries. If there is a particular pattern you think I should be looking for, happy to look.
Nothing logged at all for ‘error’, ‘warning’ or similar.