error while traversing with activated fs watcher / inotify

Hi,

reading the “depreciated” info here, I tested the filesystem watcher feature with activating the option Fs Watcher Enabled and set Fs Watcher Delay (seconds) to 60 on 4 machines (mac/win/linux) and some folders.

After enabling, I got those messages in the GUIs:

both Linux versions,

v0.14.40, Linux (64 bit) (Synology DSM 6.1.4)

and

v0.14.40, Linux (64 bit) (QNAP with Firmware 4.2.6)

throw:

Failed to start filesystem watcher for folder "foo" (foo): error while traversing "/foo": no space left on device

v0.14.40, Mac OS X Sierra, 10.12.6 (64 bit) throws: Failed to start filesystem watcher for folder "foo" (foo): watching is not supported

v0.14.40, Windows7 (64 bit) throws nothing.

On the Linux machines, there is plenty of free space, so the message is quite confusing.

The message on the Mac is very userfriendly.

No message on the windows machine means that the fs watcher is running, I guess.

Is it save to let the FS watcher active on all machines, to check, if syncthing will suport the machines with later versions? Or will it cause problems?

Huch, I wasn’t aware of this deprecation notice - I assume this will generate some momentum.

About the linux/no space problem: Please give more information about your setup like folder paths and disk size/usage. The slash in the path looks suspicious, is that the original log line?

Mac should work according to https://github.com/syncthing/syncthing/issues/4503, or aren’t our release built with cgo yet? @calmh

If everything works fine, you should get a “Started filesystem watcher for folder” message.

The current implementation is new, so there may still be issues (as you are discovering), but even if it fails, you still have the normal periodic scans to pick changes up.

0.14.40 wasn’t built that way, which precipitated the issue you linked @imsodin. So on Mac the fs watcher isn’t supported in 0.14.40. I think deprecating the old mechanism might have been a bit premature, but testing is good.

Isn’t the Linux “no space” just running out of whatever widgets inotify needs, so out of space for something but not disk space?

Yep, it is. I just checked and syncthing-inotify checks the error string while the new implementation tries to check syscall.Errno. That won’t work because that’s not whats reported by upstream. I will try to get this fixed upstream first.

@sisa Follow this advice to get your Linux devices working: https://github.com/syncthing/syncthing-inotify#troubleshooting-for-folders-with-many-files-on-linux

Hello,

Sorry just so I am clear about the depreciation notice. Does it means that from v0.14.40 syncthing-inotify won’t work and to get file watcher you need use the built in mechanism in syncthing and activate the option from advanced folder config?

Thanks

It will work, but it’s less work to use the built-in thing.

I updated the syncthing-inotify readme with some additional info and a small cautionary note. I would replace that with a clear call to switch once UI/docs are in place (which I should do…).

Thank you Simon,

I did

# cat /proc/sys/fs/inotify/max_user_watches

8192

# sudo sh -c 'echo 204800 > /proc/sys/fs/inotify/max_user_watches'

# cat /proc/sys/fs/inotify/max_user_watches

204800

to revert the setting in case the system behave strange and need to reboot… just in case.

This is my setting for on of the folders. They look mostly the same:

        <folder id=„foo" label=„foo" path="/volume1/foo" type="readonly" rescanIntervalS="604800" fsWatcherEnabled="true" fsWatcherDelayS="86400" ignorePerms="true" autoNormalize="true">
        <filesystemType>basic</filesystemType>
        <device id=„#####" introducedBy=""></device>
        <device id=„#####" introducedBy=""></device>
        <minDiskFree unit="GB">1</minDiskFree>
        <versioning></versioning>
        <copiers>0</copiers>
        <pullers>0</pullers>
        <hashers>0</hashers>
        <order>newestFirst</order>
        <ignoreDelete>false</ignoreDelete>
        <scanProgressIntervalS>0</scanProgressIntervalS>
        <pullerSleepS>0</pullerSleepS>
        <pullerPauseS>0</pullerPauseS>
        <maxConflicts>10</maxConflicts>
        <disableSparseFiles>false</disableSparseFiles>
        <disableTempIndexes>false</disableTempIndexes>
        <paused>false</paused>
        <weakHashThresholdPct>25</weakHashThresholdPct>
        </folder>   

For the Slash in the path, I copied the GUI messages, not from the Log. There is no log atm, sorry. Screenshot:

You didn’t explain if that solved the issue or not.

Restarting syncthing takes quite a while :-}

At least, no webGUI messages came up anymore on both of the linux NAS.

In the Windows logfile, I recognized the

INFO: Started filesystem watcher for folder "foo" (foo)

So I guess, if no (webgui) error messages came up on the both linux NAS, they will be fine.

I keep an eye on it and will report if anything unexpected happened.

Thank you all.

1 Like

Ok, the first “unexpected” happening:

2017-12-12 00:35:30: Failed to start filesystem watcher for folder "foo" (foo): error while traversing "/volume1/foo/folder/F/Foo & Bar": no such file or directory

This is a WebGui Message, I received on a Syncthing “v0.14.41, Linux (64 bit)” after a restart of the Syncthing service via WebGui.

The folder complained about is a folder, I recently deleted (or renamed). Maybe at the same time, the server was indexing after the restart.

While I see no harm here, I don’t understand, why I’m getting this message in the WebGui. What will it tell me I have to do?

Yeah, that’s a bug. Presumably the file system watcher is figuring out what things it should watch, and then when it tries to do so it’s no longer there. It should simply ignore that error. Mind filing this on Github? Otherwise I can do so for you.

The filesystem watcher aborts when it encounters any error while setting up. There is currently no “recovery” mechanism, meaning if the error was just transient (as I understand is the case here), it wont fix itself by retrying. What you can do is pause and unpause the folder to try again. That definitely isn’t very user friendly though. Just behaving as everything was fine and retrying periodically doesn’t feel right either. This is also related to the discussion of whether to adjust rescan intervals on failure (which was brought up in https://github.com/syncthing/syncthing/issues/4552, but maybe deserves it’s own issue).

Maybe we should try to reactivate the watcher on each scan? (and the scan interval stays lowish while this doesn’t work)

Does the watcher clean up after itself if it fails to start watching half way through like in this case?

Fwiw my filesystem watcher tries to set up the watcher periodically if something failed (it’s quite a short period, in the order of minutes) and no one has ever complained.

1 Like

How often do you scan while there is an error?

If I get an error setting up the watcher (e.g. the path doesn’t exist), I retry every 3 seconds. I don’t think I ever explicitly tell Syncthing to scan a folder that’s reappeared, just if something changes (although there might be a side-effect somewhere which cases me to tell Syncthing to rescan a folder that’s reappeared).

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.