Database updates have been batched into a “transaction” for atleast a few years, so there is nothing to try out, as you are already running it.
1 Like
I don’t know what else to add then. I’ll be glad to help in what I can within my possibilities.
I don’t think there is anything to add, if there was a way to fix this, we probably would have done that already.
Using fsyncdata vs fsync might win something, but I doubt it’s much, and you still have to fsync the directory.
if there was a way to fix this, we probably would have done that already.
Isn’t there any open discussion/ticket for golang where is requested to have a better way of syncing writes to disk that is not file per file?
It’s nothing todo with golang.
“Sync all files to disk” is something that is not portable/work on all filesystems/platforms, hence not something we could use even if it was there
Are you sure it’s not possible to implement it for all systems? I’m sure it’s on Linux (sync command) and it seems it’s also possible to do on Windows (https://docs.microsoft.com/en-us/sysinternals/downloads/sync)
Don’t know about macOS, but I found this:
Yeah, Windows has FlushFileBuffers ; you can do a whole volume but only if you have admin privileges.
FlushFileBuffers on a single handle seems to be the internal implementation of Fsync in Go on Windows … the docs warn you against using it to implement cases where the application “needs to ensure critical data is written to persistent media”, and it’s noted to cause performance issues in other apps.
I got here because syncing to my NAS was really slow though … and that’s on Linux / btrfs. Lots of disk rattling and excruciatingly slow (a few files per second) with fsync on, tens of thousands of files synced in seconds with it off.
My thinking at the moment is that the likelihood of data loss is way higher with fsync ON … because the enormous length of extra time waiting for a sync to finish is far more likely to have a power loss event in it … and I have a UPS on my NAS but not my main desktop.
Would it make sense for you to “pre-sync” a new device by copying the files that need to be synched via some faster method, such as sneakernet? If Syncthing’s security and data safety are needed, perhaps a script to zip the files, sync the archive via Syncthing, and then unzip the archive on the destination would work.