Files are never modified - new files are created, and occasionally some are deleted.
According to the UI, the first two nodes are mostly “Scanning”. The second node has ~50GB free RAM, so maybe it is keeping the filesystem metadata cached which explains how it can be scanning and yet not doing any I/O… but even with everything in RAM, it still takes several minutes of processing to detect that there has been one file added. (Does syncthing stat() every file even when the directory mtime hasn’t changed?)
Even when the scanning is finished, and the first two nodes are just sitting there in sync, idly being “Up to date”, they’re still using several cores :S
The third node is “Syncing”, which makes sense because it’s the only node that doesn’t have all the data - but 15MB/s of disk IO to 300KBs/ of network IO seems out of balance…
I’ve now mounted .config/syncthing in a ramdisk - that seems to have brought the I/O down to 2-3MB/s, but still only getting 300KB/s from the network (With all CPU cores maxed out, so I guess that was the bottleneck all along. Are index updates O(n) to the size of the repo, so when I’ve got 4 million files, it takes 300ms to update the state of each block to say that it has been downloaded, so I can only download ~3 x 128KB blocks per second?)
I’mma try setting the rescan interval to 0 (IIRC that disables scanning?) to see if things settle down if I just leave it alone for a while…
ED> Maybe if disabling scanning helps, switching to inotify or something would be a good idea?