So, I let Syncthing run overnight on both device to allow the sync state to stabilize. During this process, there were failed items, but this morning there are none. Of the 17 folders on the iMac, 3 are out of sync and 14 are up to date. All 13 synchronized to the Linux box are up to date.
Although the failures are temporary and I don’t think they should affect final synchronization outcome, I’ll briefly describe what know about them so far. I have observed two reasons for failures: too many open files and generic failure.
Generic failures happen on the Linux box. I don’t know what they mean and have not had a chance to explore them.
Too many open files happens on the iMac. It happens most frequently when Syncthing has a lot of work in progress. When failures are occurring,
lsof shows just 250-260 files open for Syncthing and about 17,000 open system wide. Since
ulimit -n shows 4864 open files per process and
kern.maxfiles is 196,608, too many open files does not yet make since to me–but my knowledge in this area limited and mostly theoretical.
Note: Below I use the terms source and destination. I realize that since both folders are send-receive, that these terms are technically incorrect. I chose these terms because the destination started empty and no changes were made during synchronization. Thus, the end effect should be the same as performing a simple copy from the source to the destination.
THE SOURCE iMac
Figure 1 shows the source folder on the iMac. There is a wide variety of files types in the folder (although this should not matter). Figure 2 shows the four out of sync files. Note the difference between the global and local states. There are 11,502 files that should be synchronized. Leaving out the 4 that are out of sync yields 11,498, which is the local state. There are also 4 ignored files (all macOS .DS_Store files); but, these should not be included in the counts.
I cannot account for the difference between the local and global state counts.
find folder -type f | wc -l confirms there are exactly 11,506 files in the source folder as expected (11,498 + 4 ignored + 4 out of sync). I don’t yet know if this difference is relevant. However, I have two other out of sync folders on the iMac; and, the counts are different on those too.
Figure 1. Source Folder
Figure 2. Out of Sync Files
THE DESTINATION Linux Box
Figure 3 shows the the destination folder on the Linux box. This folder started empty and is up to date according to Syncthing. All 11,498 files from the iMac’s local state have been synchronized.
Note: Watch is disabled in the Linux box because all changes will on other devices; thus, watch does not seem necessary.
Figure 3. Destination Folder
LOOKING FOR AN EXPLAINATION
Previously I said that I did not believe this was a bug. For now, I still believe this. However, it is not resolved by running rescan on either or both devices.
APPENDIX: DEVICE INFORMATION
The device information for iMac and Linux box are shown in figures 4 and 5 respectively.
Figure 4. Source Device
Figure 5. Destination Device
Any insight will be greatly appreciated.