In my setup, I have several machines (mostly Linux) running Syncthing, but it’s thie single OSX machine that seems to be showing problematic behaviour.
The usecase is I have a couple of directories syncing between my wife’s Macbook (running OSX) and a home server (running Debian) so that files are later processed by my automated backup system even if the wife’s laptop is off. The wife tends to work on the run, so it’s common for a new file to appear (be saved to a synced directory) immediately followed by the laptop going offline (getting to sleep mode after closing the lid). In most cases such a file never gets synced, I get “peers who had this file went away, or the file has changed while syncing. will retry later” on the home server even after the laptop becomes available again.
I cannot be sure that all such files (files created shortly before machine goes offline so that they don’t have time to sync) get stuck, but I have observed at least 20 such cases since November when I first set up Syncthing on that laptop, and I have just got another case today, with both machines running the newly released 0.14.47).
The only way to get the files unstuck seems to be renaming the file in question on the laptop and then renaming it back again. The home server then says the file with original name was modified by the laptop (not “added”, mind you, “modified”!) and happily syncs it in.
With such a long preamble, my question is fairly short: what can I do and what should I investigate before an useful bug report can be created on this? Yes, I’m totally sure this is a bug (or at least a definite “should not happen™”), but this post is definitely too long and vague for GitHub.
Maybe this is a client implementation issue (this would also be the easiest problem to solve if this is in fact the issue). Try this implementation. Have never had an issue with it.
Minimally if the problem continues to exist under a different client implementation that is good information as to what the issue actually is versus what it is not.
I use the official OSX build on that machine (with autostart as described in official docs). Giving another implementation a try looks like a sound idea.
Out of curiosity, HFS+ (pre-High Sierra) or APFS (High Sierra)? I haven’t seen the behavior myself although I Mac a lot, but the issue as we understand it happens when a file gets modified without the timestamp getting bumped. This would be more likely to happen on HFS+ than APFS due to timestamp resolution. (And I don’t see how it connects to the sleep cycle, not that I disbelieve your report.)
I’m not that versed in Macs and my usual way of handling them by opening terminal and trying the usual commands doesn’t seem to work (at least there’s no parted in OSX). But “mount” says it’s hfs, and it says “Journaled HFS+” in System Properties GUI, so I suppose that’s what it is.
Yeah, OK. That means one second timestamp resolution, and means that with unlucky timing there can be a write to a file, Syncthing hashes the file, there is another write in the same second (that doesn’t update the timestamp), and Syncthing is none the wiser. (APFS does nanosecond timestamps so this scenario doesn’t happen.)
Audrius’ patch fixes this.
Still, it shouldn’t be very common. We don’t scan that often, and even with filesystem notifications the scans are delayed a bit from the writes.
Is there any particular type of file that this happens to more often, so that it could be correlated to how that program writes the files?
I’d say 80% of cases I’ve seen were screenshots. Other cases that come to mind are downloaded JPEGs and MS Word’s DOCX files, but that’s the minority, and those 3 cases basically cover most of files ever being created on that laptop. I think I’ve never seen this happen to MS PowerPoint PPTX (the majority of work my wife does on the laptop in question is with those), but they are usually over 500 MiB, and the issue only seems to happen to files under ~150 KiB.