Is it possible to configure Syncthing to do builk updates? We are syncing code from a laptop to a remote server. When we do a big checkout/merge, files are synchronized and uppdated on remote little by little. This introduces performance issues with the remote hot reloader of our application. It would be nice to tell syncthing to update all files at the same time when all changes are available on remote. It’s done for single files using the *.tmp files, but we would need to update all *.tmp at once.
Not really sure what you mean.
The filesystem doesn’t support “bulk” operations, you have to rename files one by one, there is no other way.
Also, syncthing has to fsync every file to prevent data loss which takes time.
Lastly, syncing a git checkout is a terrible idea and you are likely to get it corrupted: syncthing - How to clean damaged objects in git repository? - Stack Overflow
1 Like
The filesystem doesn’t support “bulk” operations, you have to rename files one by one, there is no other way.
Yes, but all *.tmp files could replace their original file at the same time. Right now, they are updated as they are synched, doing many file updates in a long period of time and causing the hot reloader to restart many times.
Lastly, syncing a git checkout is a terrible idea and you are likely to get it corrupted
We don’t sync .git, just the source code
How would that work? You download all files as temp files, and then replace them all at once?
What if the files keep changing, you never replace them?
Wouldn’t the temp file creation still trigger the reloader?
Just an idea, but you could set up some kind of a script that packs the files into an archive, and then sync only that when you’re ready.
You could also try disabling fsync in advanced options. This might lead to data loss, but will speed up things for many small files.
How would that work? You download all files as temp files, and then replace them all at once?
Yes, that’s what I was thinking
What if the files keep changing, you never replace them?
It could have a time windows. If no changes are updated in X seconds, then, all files whose sync is completed, are updated.
Wouldn’t the temp file creation still trigger the reloader?
No, those files can be ignored by the hot reloader
You could also try disabling fsync in advanced options. This might lead to data loss, but will speed up things for many small files.
Yes, we are already doing that. It improved the IOPs behavior, but not the hot reloader.
Right, this is what we tried first. But we have a complicated .stignore and it’s hard to apply the ignore logic when we do the bulk update.
I suspect you’d be better of talking to the API. Subscribe to folder events, when it changes from “syncing” to “idle” you’re in the state you want.
Yeap, that’s what we were planning. We can also track which files were synched, and just copy those ones