Newly created file synced fast from windows to linux, but not vice versa

Support
1

syncthing version: 1.23.6

devices: win11, centos7

I changed the fs watcher delay from the default 10s to 1s and enabled watching on windows and linux. Anything else is default setting.

File modification works as expected, syncing in real time, either from windows to linux, or linux to windows.

File creation is not synced that fast. When creating new files (.txt, .png) on windows, the created files immediately appear on linux. However, when creating new files on linux, the files would not be on windows, even after 3 minutes. I haven’t tested for how long it would need to sync. For now, I just manually rescan on web-GUI.

At first, I thought it was the inotify limit problem as described at FAQ. However, I checked the log at stdout on linux, everything is fine.

Do you have any ideas what causes the delay from linux to windows when creating new files?

2

On the Linux side, try lowering Syncthing’s “Full Rescan Interval” setting for the folder from the default 3600 seconds to 30 seconds.

If a new file appears on the Windows side shortly after 30 seconds, then the issue is with inotify.

What’s your current setting for /proc/sys/fs/inotify/max_user_watches?

How many files and directories are in your problematic Syncthing folder?

Keep in mind that the limit is the sum total of all watches per user per inotify instance.

1 Like
3

Nit all filesystems support watching. Make sure the folder on the linux side is on a sensible filesystem. Not some fat32 thing, or some network attached thing, or some custom nas filesystem.

4

Thanks. Yes, you are right. After changing rescan interval from 3600 to 30 seconds, the newly created file appears on windows about 30 seconds later.

My current setting for /proc/sys/fs/inotify/max_user_watches is 1048576

My Syncthing folder has 1336 folders and 16932 files.

Can I further decrease the rescan interval? Would it use a lot of cpu?

5

What type of storage is your Syncthing folder on?

And the volume format? (e.g., XFS, ext4, btrfs, SMB, NFS, etc.)

That’s more than enough given the amount of files you’re syncing.

Besides max_user_watches, there are also two others:

/proc/sys/fs/inotify/max_user_instances

/proc/sys/fs/inotify/max_queued_events

max_user_instances is usually 128, which is plenty unless a user is going to be running a lot of apps that are requesting file/directory watches.

max_queued_events is usually 16384. Also usually not a problem except perhaps when there’s a lot of file/directory activity. If the queue overflows, event notifications are dropped.

Without knowing anything about the hardware involved, I can only guess, but with under 1,400 directories and 17,000 files, it shouldn’t be too bad using a lower rescan interval.

However, it really depends on the storage medium, type of drive interface, what else the system is tasked with doing (e.g., if it’s a file/web server), etc.

The best option is to get the file watcher working.

1 Like
6

Actually, I don’t know what the type of the storage is. I work on a linux cluster, which have several nodes (computers), node01, node02, etc. But from every nodes, I can access the same file, e.g. /home/yann/data/, I suppose these nodes all share the same hard-disk.

I ran df -T, and find out the file system type is nfs4.

/proc/sys/fs/inotify/max_user_instances is 128 /proc/sys/fs/inotify/max_queued_events is 16384

I do not have too much apps running, I only run matlab and vncserver.

Each node has at least 200+GB free memory, and 60+ cpu. It is used for large computation. Maybe it is powerful enough to scan every 30 seconds. Nevertheless, it is better to have the watcher working.

7

Hmmm… NFS definitely makes things a trickier. Especially if you don’t have any control over how the nodes and NFS are configured.

Generally speaking, file system event notifications are limited to local access of files and directories. However, it doesn’t automatically mean that network file shares are unsupported. It depends on how the remote filesystem is presented via the VFS (aka., “virtual file system”) layer in Linux.

Linux’s inotify requires that a file/directory must have an inode. That shouldn’t be a problem for NFS.

Next, see if the command inotifywait is available (it’s part of the inotify-tools package that’s part of practically all Linux distros).

If inotifywait is available, point it at a file/directory that Syncthing is watching like so:

inotifywait /my/syncthing/folder/some/file.txt

Then in another terminal, touch the file inotifywait is monitoring.

  • If the test succeeds, inotifywait will print a message about what filesystem event was seen and then immediately exit.
  • If the test fails, then you’ll have to review how the NFS share is configured and also repeat the test on a file that’s not on a NFS share to help rule out a system configuration issue.

I’ve tested various remote filesystems including NFS, SMB/CIFS, – even SSHFS – and inotify has worked fine on newer Linux kernels.

Of course, YMMV, depending on exactly which version of CentOS 7 is being used (Linux kernel 3.10 isn’t the latest, but it’s also getting a bit behind) and other settings.

With those system specs, RAM and CPU is unlike to be an issue (except perhaps if the system is running full tilt).

8

Unfortunately, the command inotifywait is not available. Maybe that is the reason why watcher does not work?

cat /etc/centos-release gives Rocky Linux release 8.8 (Green Obsidian). Seems it is centos 8.

9

inotify is baked into the Linux kernel, so the lack of inotifywait just means the inotify-tools end-user package isn’t currently availale (it’s an optional install) rather than being the reason for the issue.

Since Rocky Linux is derived from the corresponding version of RHEL, and RHEL is downstream of CentOS, they basically share the same Linux kernel somewhere down the line.

I don’t have a Rocky Linux system handy, but I do have access to an existing CentOS 8.x system with a 4.18 kernel (same as in Rocky Linux 8.8) and don’t have any issues with inotify.

You’ll have to ask that the inotify-tools package be installed, compile it from source, or find another way to test inotify support for your particular system configuration.

As was already mentioned, filesystem event notifications don’t work on every type of filesystem, especially ones that are pseudo-filesystems.

10

Ok, I’ll try to compile inotify-tools package myself then do the tese.

11

I tried inotifywait /my/syncthing/folder/some/file.txt

After I touch /my/syncthing/folder/some/file.txt, inotifywait says /my/syncthing/folder/some/file.txt OPEN. I assume this means inotify works.

By the way, I think we may miss an important detail in my very first post, where I said the file modification on linux can be immediately shown on windows. This might already indicates inotify works. I do not know file creation cannot be immediately shown.

12

Yes, your assumption was spot on. :slightly_smiling_face:

Its sibling, inotifywatch, counts filesystem events and prints out a summary when the key combo [Ctrl + C] is used.

So the good news is that inotify is definitely working, and you just need to figure out why Syncthing isn’t receiving notifications when there’s a create event.

Now have inotifywait watch a directory under your Syncthing folder while you create a file in it from Windows, e.g.:

inotifywait /my/syncthing/folder/some/

In theory, the directory’s modification time will be updated when you create a new file in it (which you had said hasn’t been an issue).

No, I got that part. :nerd_face:

Not all filesystems support all available filesystem event types (or some might involved delayed notifications).

Do file deletions and/or renames on Linux also take a long time to sync?

13

I tried watch a folder and created new files both on linux and windows. Both gave the following result.

Setting up watches.
Watches established.
/cluster/Syncthing/tmp/ CREATE new file

The delete and rename on linux can be immediately shown on windows.

At last, I just found out this behavior (linux creation cannot be reflected on windows) is not 100% reproducible?!!! And now I can create a file in linux, and this file immediately shows on windows.

So, it seems sometimes sync in both direction works well. But sometimes the linux to windows doesn’t work. I suspect this maybe related to available number of system resources at that time, e.g. watchers. Perhaps last time it just ran out of watchers so I have to wait.

14

You mentioned earlier that you had close to 17,000 files in the Syncthing folder. If it’s a source code tree, perhaps all of the files are being touched by some process such as an IDE, revision control, etc.

Since your /proc/sys/fs/inotify/max_queued_events is set to 16384, I can see how there might be intermittent times when inotify’s event queue is overflowing (inotify will simply drop event notifications when that happens because they are considered low priority).

15

If it’s a source code tree, perhaps all of the files are being touched by some process such as an IDE, revision control, etc.

Indeed! I used matlab most of the time, which can be thought as an IDE. After I watch I folder using inotifywait /my/syncthing/folder/some/, if I switch to the window of matlab, even I did not create or modify file, inotifywait would stop, saying

/my/syncthing/folder/some/ OPEN,ISDIR

I forgot to mention this in the last post, I thought this is not important.

16

Try repeating the test with the command inotifywatch /my/syncthing/folder/some (also part of the inotify-tools package).

Switch to Matlab, then hit [Ctrl + C] in the terminal where inotifywatch is running and it’ll output a summary like so:

$ inotifywatch testdir/
Establishing watches...
Finished establishing watches, now collecting statistics.
total  attrib  close_write  open  create  filename
12     3       3            3     3       testdir/
17

./inotifywatch /autofs/syncthing/tmp

Establishing watches...
Finished establishing watches, now collecting statistics.
total  access  modify  close_write  close_nowrite  open  create  filename
232    95      4       3            63             66    1       /autofs/syncthing/tmp/

I only created a file and modified a file. The summary shows much more operations than I did.

So, if the IDE takes more operations that uses most of resources, what should I do to avoid this? Maybe I should move some files that do not change too often out of the syncthing folder?

18

You mentioned having 1,336 directories and 16,932 files under your Syncthing folder. Since inotify doesn’t recursively watch subdirectories, that’s a minimum of 1,336 watches. It’s well under the 1048576 limit for your system, but then there’s max_queued_events, which is 16384.

If just one directory in your test generated 232 events, you’ve got enough files and directories to make it easy to overload inotify’s queue.

Assuming that you don’t have enough access to change the system settings (otherwise you could have just installed inotify-tools directly from the Rocky Linux repo), your options are limited.

A couple of workarounds to try:

  • Set Syncthing’s fsWatcherDelayS to 5 seconds instead of the 10 second default. Syncthing will wait 5 seconds after being notified of a filesystem event before scanning for changes.

  • I haven’t looked closely enough over Syncthing’s code to know for certain it would work, but since Syncthing is multi-threaded, using multiple smaller directories might get around the limits on your system.

If neither workaround above improves things, not syncing files that seldom or never change is likely the only solution.

1 Like
19

You mentioned having 1,336 directories… . Since inotify doesn’t recursively watch subdirectories, that’s a minimum of 1,336 watches

I should clarify that these 1336 directories are all the folders in my syncthing folder, including subfolders of subfolders.

I should also clarify that in the following test, the /autofs/syncthing/tmp only have files, there is no subfolders.

./inotifywatch /autofs/syncthing/tmp
Establishing watches...
Finished establishing watches, now collecting statistics.
total  access  modify  close_write  close_nowrite  open  create  filename
232    95      4       3            63             66    1       /autofs/syncthing/tmp/

By the way, if syncthing does not watch sub-folders, how can it know contents in sub-folders changed? Only by rescan? My folder structure looks like

/autofs/syncthing/code/
/autofs/syncthing/data/
/autofs/syncthing/result/
/autofs/syncthing/tmp/
/autofs/syncthing/a.txt
/autofs/syncthing/b.txt

If inotify does not watch sub-folders, only changes in a.txt and b.txt can be watched, right? For any changes in sub-folder /autofs/syncthing/code/, I have to rely on the rescan to do the sync, right?

I already set the fsWatcherDelay to 1 second to accelerate the sync.

20

Got it. It was also what I was assuming you meant. :slightly_smiling_face:

If /autofs/syncthing is the root of the Syncthing folder, it didn’t make any difference for /autofs/syncthing/tmp.

No, Syncthing does watch subdirectories, but inotify requires separate watch requests for each one – i.e., simply asking inotify to watch /autofs/syncthing isn’t enough because inotify won’t descend into subdirectories.

So if /autofs/syncthing contains 1,336 subdirectories, then a minimum of 1,337 watches are required (1 for /autofs/syncthing + 1,336 for the rest).