Indeed, I can reproduce this on any filesystem with:
mkdir /tmp/test
cd /tmp/test
# create 2 directories
mkdir dir1 dir2
# create a dummy file
touch dir1/file
# create hard link to file in second directory
ln dir1/file dir2/file
# watch first directory
inotifywait -m dir1 &
# modify file from second directory
echo test >> dir2/file
# no event is generated...
For the curious, the rationale for this (intentional) behavior is described here: https://lwn.net/Articles/605128/
I have done a quick read of the Syncthing code around inotify, and here is what I found:
At some place the code seem to detect and properly handle internal (go channel) overflow of events, and act accordingly by triggering a full rescan, ie: https://github.com/syncthing/syncthing/blob/1e71b0093626a00867725813bebc733e967f5353/lib/fs/basicfs_watch.go#L77-L78 However in at least one other place it seems to be ignored: https://github.com/syncthing/syncthing/blob/1e71b0093626a00867725813bebc733e967f5353/vendor/github.com/syncthing/notify/watchpoint.go#L90-L91
The native inotify overflow events (IN_Q_OVERFLOW) are silently ignored: https://github.com/syncthing/syncthing/blob/1e71b0093626a00867725813bebc733e967f5353/vendor/github.com/syncthing/notify/watcher_inotify.go#L272-L274 So the event is not propagated and a full rescan can not be triggered.
I am not familiar with the Go language or the Syncthing code base, so I may have wrongly interpreted the code.
Do you confirm my interpretation of the code, and are these issues fixable?
More or less: You did not do the distinction between actual syncthing code and notify code. The latter is a separate library, that we maintain a fork for which is identical with the original https://github.com/rjeczalik/notify except for a few modifications we need. The notify library does send events without blocking, i.e. drops them - that’s a design choice. That’s why we use a buffered channel, so this never happens, unless this channel is full, which we detect (as you found out correctly). So that part is fine.
Inotify overflow events: Looks they are dropped, maybe that could be fixed. It’s in the notify library too, so you’d have to ask there, but I guess it’s not an issue because they send events without blocking, i.e. fast, so overflow shouldn’t be an issue. Just thinking out load here, I don’t know how inotify actually works.
Yeah I saw https://github.com/syncthing/notify but got confused since what is in https://github.com/syncthing/syncthing/tree/master/vendor/github.com/syncthing/notify is not a submodule pointing to this. Since you say it is the same, I suppose this is just a copy of the repository.
At least in theory it should be possible that the kernel pushes lots of events in the inotify queue, but the user space Syncthing process does not get scheduled often enough to consume the events (because the CPU is overloaded or because the process priority is very low) until the queue is full.