I use Syncthing to synchronize downloaded files across several computers. Some of those downloads are via http, some via ftp, and some via bittorrent. Unfortunately, this causes issues with Syncthing, as it starts syncing files as soon as they start downloading, then keeps re-syncing them as they change over time, until they are finished.
This results in:
constant re-hashing as the files download
bandwidth being used to upload the files while they are being downloaded
disk access time being taken up due to #1 and #2
Now, itâs certainly not a critical bug, but when Iâm spiking activity, it can easily end up in the downloading computer thrashing as itâs simultaneously downloading dozens of files, rehashing them constantly, and uploading them constantly â as well as the other computers constantly getting a âfile has changedâ error as theyâre trying to sync said files.
A workaround, which is the feature that I am proposing, would be a âlazy hashingâ option. Essentially, if you enable âlazy hashingâ, you can enter a time period (5 seconds, 30 seconds, 5 minutes, whatever). Syncthing will then skip hashing a file that has been modified more recently than that. Basically, it will skip (and ideally queue) any file that has been added/changed, until it hasnât been modified for X timeperiod.
This avoids breaking the normal flow of how Syncthing works, while accounting for files that are constantly changing (due to downloading, streaming a filecapture to disk, etc). Anyone who doesnât want to use it, just doesnât enable the âlazy hashingâ feature, and itâs no skin off their backs.
This sounds like a syncthing-inotify thing. Potentially it could learn that when a file is continuously changing it should start slowing down the rescans.
You can already solve your problem with existing features.
Ignores: Most download managers can add an additional file extension to the in progress files. If you ignore files with that extension, Syncthing will scan it only after the download is finished. Chrome uses such an additional file extension for downloads as well.
REST API: You can set the scan interval to 0 and have you download manager (or manually) send Syncthing the command to scan that specific file after the download is complete.
Ignores: Some can, yes, however at least in my case I donât believe that the torrent client that I am using can. And Iâm unaware of any pure FTP clients (not download managers with FTP built in) that do that.
REST API: This wouldnât address people using inotify, requires programs that can send syncthing commands on download completion, and would preclude other files (made by other programs) from being synced, or necessitate splitting downloaded files to one folder and non-downloaded files to another.
Curiously enough Syncthing actually had this feature way back in version 0.2 or thereabouts. I remember it caused confusion and frustration when it didnât rescan files that had changed and getting the threshold anywhere near right was difficult. Iâm not super fond of going back to it.
I donât know what torrent client youâre using, but some of them (e.g. deluge) have a feature where it will move completed downloads to a different folder. So you could have your completed downloads folder synced with syncthing, but not the incomplete downloads folder.
Yes, thatâs what Iâll end up doing for the torrents. Unfortunately, rTorrent is a pain in the bum to do it with (you have to set it individually for each watchdir).
Unfortunately, a lot of the activity is automated â flexget parses rss feeds for what I want, then downloads the .torrent into an rTorrent watchdir, automatically starting it. I guess maybe I could use a daemon to pause syncthing when rTorrentâs CPU usage spikes, but that would be a whole lot of work, as Iâd have to learn how to do all of it.
I see the âarguments againstâ make sense, but I do also experience the same issue as the OP. Especially because in my network there are several NASâses, that really keep quite busy with the syncing due to this. Itâs a network with 12 people, files change all the time and are of all kinds, so temp switching off or relying on excluding file extensions wonât work.
Maybe somebody will come up with an alternative approach; for me it certainly wonât be a useless feature. Itâs actually the only inconvenience I still have with Syncthing; other than that, Iâm extremely happy with it.