Recently, I have encountered a few times a problem when one of the folders simply stopped detecting new changes. This is not common and seems to happen rarely, but once it does, only restarting Syncthing seems to fix the issue.
Unfortunately, I don’t have any debug logs to actually provide more detailed information, so this is more of a question than a proper support request. Do you have any idea what could cause potential file watching issues when it comes to Windows?
As for my configuration, I can only say that I have a lot of folders added to Syncthing. Some of then contain a lot of files that change very often. Are there any known limits in Windows regarding the number of file watched items, as it is the case in some other systems?
The issue has been happening in Windows 10 running Syncthing v1.18.x (custom built). The filesystem is just the default NTFS.
Right now everything is working properly, but when I had the issue for the last time around 2 days ago, pausing and unpausing did help, but only in a way that it triggered a rescan, which then detected the modifies files. File watcher still didn’t work for the folder later on though. I could easily know it, because the last scan date was not changing, even after modifying plenty of files. Only a manual re-scan forced them to be detected and synced.
This is kind of a big problem for me, because I normally set the rescan interval for my folders to 86400, as I just have too many of them to keep them being scanned every hour, but with a broken watcher this also means that changes may be delayed up to 1 day, which is very bad .
Just for the record, I couldn’t find anything useful in the Event Viewer either, and also the files themselves have actually changed content and/or mtime too, and the watcher does detect them under normal circumstances with no issues.
Not much clue. The fact that a folder restart (unwatching and watching the folder) doesn’t help, but restarting syncthing does, implies a problem with some initialization code in the notify library. But there isn’t any initialisation as far as I can see. So maybe a deadlock. Without a reproducer that’s going to be very hard to find. The env var NOTIFY_DEBUG enables debug logging in said library, maybe that would help if you can capture logs when it happens.
Copy the folder physically to a different location on the disk.
In the copied folder, add/modify/delete some files.
Delete the original folder (on the disk), and immediately replace it with the copied one.
Syncthing fails to detect the changes. No future changes are also detected in the folder (until restart).
Would this qualify as a bug?
Edit 1:
The reproduction steps are overly complicated. Just renaming the folder to something else, and then renaming it back also seems result in the same issue.
Thanks for the info. Linked this thread from the issue. As I wrote there, I do intend to look into this, but can’t make any promises when. Those notify issues are usually not entirely straight forward.
I try not to, but things happen, e.g. when the folder is located on an external USB drive that has been unplugged (even by mistake), and then re-plugged. If the operation takes a short enough time, Syncthing will not notice the folder being gone, and will not stop the synchronisation either. The watcher will stop working though, with no visible errors.
Wait, I was still under the impression that restarting the Syncthing folder didn’t help. If it’s “just” a case of moving away the root folder and moving it back, without Syncthing noticing (which means you need to be really fast, faster than filesystem notifications), then yeah things breaking is expected as Jakob wrote. I am not sure whether we even get notifications for the root path, I have never thought about that case honestly.
I have done more testing today. The issue seems to be a little more complicated than I thought before.
I have tested two cases regarding the root folder.
a)
In Syncthing, add a new folder test1 with file watcher enabled.
On the disk, rename test1 to test2, and then from test2 to test1 again.
File watcher keeps working correctly.
b)
In Syncthing, add a new folder test1 with file watcher enabled.
On the disk, rename test1 to test2.
On the disk, create a copy of test2, and rename it to test1.
File watcher stops working.
Pausing and resuming the folder doesn’t help.
Watcher starts working again only after restarting Syncthing.
It seems that renaming the folder back and forth doesn’t break the watcher, but creating a new copy of it, and then replacing the original does. I have no idea on how the internals work, but it seems that file watcher is tied to the physical folder itself, regarding of its actual name/path.
I have decided to do one more test, just to confirm my assumption of file watcher being tied to the original folder.
Following from point 3 in b), if you add/modify/delete files inside the test2 folder (which is the original test1 folder renamed), this will actually make Syncthing scan the test1 folder. Basically, file watcher will be triggered by changes in test2, which will then make Syncthing scan test1 and this way detect changes there. Fortunately, it doesn’t seem to actually add any of test2 changes to test1, so I don’t think that there is a security risk here. Still, I would say that the behaviour is at least quite interesting .
Not sure if of any use, but this is a DEBUG log of what happens after adding a new text file to test2, which then triggers a scan of test1 (which itself is empty).
20:15:02 Enabled debug data for "watchaggregator"
20:15:08 aggregator/"test1" (3pxhf-ibfjg): Tracking (type non-remove): New Text Document.txt
20:15:08 aggregator/"test1" (3pxhf-ibfjg): Resetting notifyTimer to 10s
20:15:18 aggregator/"test1" (3pxhf-ibfjg): Notifying about 1 fs events
20:15:18 aggregator/"test1" (3pxhf-ibfjg): Resetting notifyTimer to 10s
20:15:18 sendreceive/3pxhf-ibfjg@0x131ae1e0 Scan due to watcher
20:15:18 sendreceive/3pxhf-ibfjg@0x131ae1e0 scanning
20:15:18 3pxhf-ibfjg Snapshot()
20:15:18 3pxhf-ibfjg Snapshot()
20:15:18 3pxhf-ibfjg Snapshot()
20:15:18 3pxhf-ibfjg WithPrefixedHaveTruncated(7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4, "New Text Document.txt")
20:15:18 sendreceive/3pxhf-ibfjg@0x131ae1e0 finished scanning, detected 0 changes
20:15:28 aggregator/"test1" (3pxhf-ibfjg): No tracked events, waiting for new event.
The only thing that is surprising to me here is b.5: That should remove the watch and readd it on the newly created directory. The rest is expected: We tell windows to watch test1, and that’s what it does even if you move it somewhere else. I don’t think we need to cover that scenario, changing the syncthing root folder shouldn’t be normal. However it can happen, so it should work if the folder goes into error state and comes out of it or after a folder restart, as both of those restart the watcher.
Yeah, the reason why I did that in the first place was because I was doing mass changes in the folder, and I didn’t want to do them live, so I made a copy of it, made the changes there, and then I just replaced the original with the copy. I didn’t think much about it, and the whole operation seemed to proceed smoothly. Only a few hours later did I realise that the folder had not been detecting changes anymore. I think that at least adding a note to the docs may be useful though.
I can confirm again that neither restarting the folder or coming out of error state fixes the problem . Restarting Syncthing appears to be the only solution right now, or otherwise the watcher seems to keep tracking the old folder only.
Just a wild guess, but could it be that the inotify library somehow caches the watcher handles it got from the OS by path name? That would explain the connection to a moved physical folder when re-requesting the same path. Shutting down the Go process would then probably lead to the cache being discarded.
Just checked whether the problem I was having also happened in Windows, and it does not. Just posting here in case this problem is somehow related.
I also think I may have explained the problem badly in my github issue. In my case the root folder never changes, either on the drive or in Syncthing. The problem, which I can reproduce everytime on Linux is:
I have a folder in syncthing called “TESTFOLDER” with watcher enabled.
I create a directory in that folder called “TEST1”, which Syncthing picks up
I then delete directory “TEST1” and then recreate “TEST1”… on both events, Syncthing picks them up just fine.
But… from that point on, anything added to the “TEST1” folder is not picked up.
If I then rename the “TEST1” folder to something else, the watcher begins to work again. Renaming it back to “TEST1”… it stops again.
To confirm, If I do the same on Windows, there is no problem and this works fine.
It is, or at least it should be. The version in the go.mod does have the fix (the newest is just a merge commit with upstream to get unrelated changes).
.
.