I’m getting a weird error syncing between a Windows 11 machine and a Fedora 39 one. Basically it’s a ~60G (currently) photo spool archive: I dump a bunch of photos on the Windows machine (mostly) and they sync-over to the Linux one. After a certain number or volume of files are synchronised (I’m not certain which), the Linux machine reports “Out of sync” with the logs indicating it’s trying to create some “.tmp” files and getting a “read-only file system” error. But the filesystem is still mounted RW and I have tested file create and write capabilities in the directory in question as the daemon’s user. I’ve tried a bunch of “cargo cult fixes” (I.e. make a change based on a theory, rather than evidence), but none have panned out in the end.
Here’s a list of things that I thought might be the issue (or may have been for a while, but I’ve fixed):
- Spooky filesystem weirdness (FS was originally BTRFS, but I’ve switched to EXT4 for simplicity)
- Target filesystem full (Never: 100G Linux FS has 37G free)
/home
partition full (It was about 80% full, but I’ve expanded it to have about 44G/90% free now)- Weird permissions mismatch between Windows and Linux (There was one warning about this in the past, but I’ve selected the “Ignore permissions” options in the GUIs at both ends and all the other “Sync/Send” “Ownership/Attributes” options are unset.
- SELinux gotcha - Tried both
restorecon -r
and asetenforce 0
followed by a rescan on the Linux side. No effect seen. - File ownership problems - as root, I’ve chown-ed the synced tree to the user running the daemon
- “Turn it off and on” - I’ve tried restarting the Linux daemon and the VM that runs it. Hasn’t made a difference.
- Somehow the iNotify max watchers limit reached (but that should be its own, descriptive error anyway). There are other synced folders here, but the net total is less than 20k and the limit is currently 242166.
- Maximum size of synced data? (I’m reaching here) - Currently the system manages about 60G of data across 3 shared folders.
- Synced file is too large? No file is anywhere near the free capacity of the target volume.
- Try removing and re-scanning each end. No change in the end.
Here is the relevant sequence of log lines:
2024-11-13 01:59:08 Ready to synchronize "Photos" (ytp2g-gpsu7) (sendreceive)
2024-11-13 01:59:08 Unpaused folder "Photos" (ytp2g-gpsu7) (sendreceive)
2024-11-13 01:59:12 Completed initial scan of sendreceive folder "Photos" (ytp2g-gpsu7)
2024-11-13 01:59:12 Puller (folder "Photos" (ytp2g-gpsu7), item "2021/PXL_20210924_094434208.jpg"): syncing: finishing: opening temp file: open /usr/local/share/photos/photos/2021/.syncthing.PXL_20210924_094434208.jpg.tmp: read-only file system
2024-11-13 01:59:12 "Photos" (ytp2g-gpsu7): Failed to sync 1 items
2024-11-13 01:59:12 Folder "Photos" (ytp2g-gpsu7) isn't making sync progress - retrying in 1m0s.
So the 100G Ext4 photos filesystem is mounted at /usr/local/share/photos
and then there’s another “photos” sub directory there that is where Syncthing does its work. As I stated before, this filesystem has 37G free. There are no files anywhere near that size (the one in the logs above is 2.5M)
I’m not sure what the size/number limit is, but I’ve just noticed the sync works for a while and then eventually this error happens and won’t go away. So far it’s only present in this one shared folder.
Any clues or hints as to what the problem might be and how I might diagnose further or fix it?