I notice that there are a number of topics involving the above error message. Some of them turn out to be related to filesystem errors, others to corrupted index database - it seems highly variable.
In my case it is pretty clear that syncthing can’t copy files open by some other process. All references in the logs point quite clearly mainly to sqlite and other index files used by Thunderbird or Firefox. So no mystery there. My questions are:
Since I use syncthing as a form of periodic backup, and Thunderbird and Firefox need to run pretty much continuously on my machine, isn’t there a way that syncthing could be made to cope with files open by some other process?
If the above is not possible, could syncthing at least manage to skip these files and carry on - as at the moment, it seems that it gets stuck in an infinite loop when it stumbles over one of these files and it makes no more progress - while it keeps on filling the log with errors.
More detailed error message from the log (involving a temporary log used by an openvpn process):
Puller (folder “home”, file “/openvpn/easy-rsa/keys/openvpn-status.log”): pull: no available source device
[5JC4P] 14:00:38 INFO: Puller: final: no available source device
[5JC4P] 14:00:38 WARNING: Folder “home” isn’t making progress - check logs for possible root cause. Pausing puller for 1m0s.
No. If they’re locked, we can’t access them. Also, these are probably something like SQLite databases which on Windows also have the nasty habit of changing contents without the modification time being updated. There’s no way to handle this gracefully from the outside.
It does, essentially, but it will continue complaining to make you aware that it’s not fully doing the job you’re asking of it. The files that it can sync are still synced, but it will retry and retry forever on the ones it can’t. You’ll need to either ignore these files or let Syncthing sync them when they’re not locked. Note though that due to the issue above (files altered, modification time not updated) Syncthing may not discover that they have changed and the sync may still fail.
In general, backing up open database files is an unsolved problem. It requires cooperation of the database engine.
Yes - that’s what I sort of assumed based on the capabilities of other backup software - backing up locked files is a no go.
Regarding the second point, it is good that syncthing carries on transferring the other files. It’s just that the log and the interface only keeps on saying “Folder “home” isn’t making progress” - which would suggest that syncthing is stuck. I’m glad it isn’t stuck - but maybe the error message should be changed slightly to reflect what is really happening.
Yeah, that whole thing with the error messages is crap. What actually happens is that it does complete everything it can complete, then when all there is left is stuff that fails consistently for whatever reason it complains about “not making progress”. Because at that point it can’t. The list of unsynced files in the GUI should reflect this btw.
Maybe an per folder option “deep scan” would help.
So Syncthing would always scan and rehash all files fo figure out if they are changed… (like in the initial scan)
this could also be configured for certain filetypes/filenames
I like the sound of that. I don’t know how feasible or difficult it would be to implement, but it sounds like it would overcome the issue of some files’ dates and times not being modified when the file is.
but it will retry and retry forever on the ones it can’t
This is good that Syncthing will continue to retry sources that have gone missing, but in my tests, it keeps generating several “no available source device” lines per second in the log file. This is going to fill my var directory pretty quickly if left unmanaged. Is it possible that the log file could say something to the effect of “I’m not going to log this file anymore until it comes back online” and stop spamming?