Hi,
first of all thanks for the great effort and great tool.
I was just wondering if there is a way how to speed up incremental sync.
My use case:
Laptop, repo 120k items ~ 580GB, setup as Master, mirrored to ARM NAS, 512MB RAM (slow calculator, I know)
Syncing my Outlook folder - 14items, ~15GB takes a long time altough this is just an incremental sync. I can see that almost the whole pst file is “Copied from original” but this process takes a long time, probably because of the NAS limitations.
Another issue I had was to sync my virtual machines folder - 400files, ~250GB. This didn’t work at all, RAM went to >1GB and CPU IO wait to 90% for 48hrs. Workaroud to create folder per virtual machine - 10 files, ~20GB works well and is able to sync, however again the incremental sync of 20GB file takes a long time.
Any thoughts? Is it possible to change the block size to compare large files somehow faster?
No it’s not possible. Changing the block size most likely wont help, as you will still have to hash the whole file, regardless of the block size, but it would help cut down on memory usage.
Thanks. Well it was worth asking. I just thought that when you index a file in say 128kB chunks and then have that index and file on 2 machines, once file on machine A grows by 1MB all the previous chunks are the same so they dont need to sync. So I thought it might be enough to add the remaining chunks to the file on machine B and that’s it. But Im an engineer, not coder so it is probably not that easy as it sounds;)
That part sounds (and I am not getting tired of repeating this) like the ping-timeout bug. You need to increase the pingTimeoutS key. Since 0.11.15 you can acces it via the “advanced” settings in the gui.
Well I’m at 600s with the pingidle and it still did not help. For me it is strange that syncing 500GB of very large files in one folder is a problem but when they are split across multiple folders it all works well. I don’t know why syncthing handles those cases differently but perhaps there is a reason for it.
Hmm, is it PingTimeOut or PingIdle that needs tweak? I have idle at 600 and timeout at 120.
timeout is the important one, idle is the time between pings and timeout the time syncthing waits for a answer. And since you say that you see “copied from original” you have the GUI open? you should leave that closed because it also needs CPU, better just watch from the faster device if anything happens.
you can also look at the logs to see if the connection does time out or if everything is just slow.
We could probably benefit from a variable block size at some point, there are just some trickiness around it. I’m not sure why there would be a difference between having the files in one folder as opposed to several, though…
Note that even syncing a small change into a large file means we make a copy of the large file, so that’s an IO heavy operation even if we don’t transfer a lot of data over the network.
Well, I don’t know either but after 48 hrs of hard disks “smoking” I was getting nowhere. Both nodes had identical files but it did not work. As soon as I split it into folders it was ok. Im happy to test it for you but I’m not sure how.
Is this syncthing feature? I mean is there another way how to do it without copying the whole file? Still, even with copying, my read/write HDD speed is just around 8MB/s, which seems extremely slow. So now 20GB file takes around 2hrs t increment-sync, with minimal data over neetwork. CPU @50%, 50% IO wait, load ~3
ok, I will try that. You are right, devices were trying to sync (I assume to just exchange indexes since files were identical) but NAS timed out every 10-20mins.
Yes, I have GUI open because Im still testing and monitoring the behaviour. But during 48 hrs sync-try it was off and still didn’t help. Perhaps ping timeout will help. I have one more folder to try - 500GB of photos so let’s see
Not really, as if we operate on the original file and run into any kind of error along the way (network disconnects after fetching two changed blocks out of five) we’ll have trashed the file.
As I understand it you are using renaming [although called overwriting] as a non interruptable transaction. Fine with me and perfect for small files.
Question is: Are there no better ways to be transaction- save?
I’m not sure about the exact features of journaling file systems and AFAIK there is no standard involved - not so far.
But at least it should be possible to implement a comparable behaviour right into syncthing. With the downside of having intermediate corrupted files of course.
But when it comes to logically related files - like those of a VM - every sync intermediately corrupts them. And using the recommended process is intended to increase speed and thus should reduce the time of logical corruption.
Of course this should not be applied to small files. And the benefit would only be with incrementally growing files. But due to the issues we are facing here most likely others will avoid rewriting their huge files as well.
To fully avoid them - even temporary - I have none.
If Journaling- FS/OS had an API supporting the required features that would be the point to start at. But a short search on the web didn’t look promising to me.
Thus my suggestion is to allow temporary corruption if:
Copying the file will take significant resources
find out by setting or measures
New data is appended at the end.
Extendable by: appended almost at the end
Solution would then be straight forward:
Store a marker with original file size
Append the data and delete the marker
possibly create the delta-file as a new file to minimize corruption and append it in the end
On error crop the file back to it’s original size
Btw:
For those huge files an opt-in setting to re-index them only if the fingerprint changed should also significantly reduce IO.
Yeah… I could see an option similar to rsync --inplace, i.e. we skip the temporary if the user tells us corruption is fine. The rsync manual says
WARNING: The file’s data will be in an inconsistent state during the transfer (and possibly afterward if the transfer gets interrupted), so you should not use this option to update files that are in use.
We have the extra caveat that an rsync transfer is user initiated (so they know about it) while Syncthing does stuff behind your back. Also that if syncthing does crash in the middle of an update (or run out of disk space, etc), we may later detect those changes as something done by the user and propagate the broken file to other devices. We might take care to avoid this by having a marker for “is being modified” and so on, but who knows.
I could possibly see this as a more granular setting as well, something like “modify in place if the changes are less than [some percentage]% of the file size”. So a changed block in a VM image would be written in place, while a resaved JPEG would do the usual dance. Pull requests welcome etc.
In general Syncthing takes the approach that slower but safe is better than faster and unsafe. This applies to the choice of crypto, hashing, file handling, and so on. That should still be the default.
Yeah, inplace is the name of the game.
This extends my first suggestion [to append only at the end of files] by some kind of binary patching. Guess that’s what syncthing somehow already does on those temporary copies(?)
That should clearly be handled by the “journal”- lock.
As long as a file is locked it must not be propagated but rolled back (and be unlocked right afterwards).
A problem is on the other side: If the user changes the file after a crash but before sync was able to roll it back we’d have a mess.
But I’d have no problem with “locking the user out” until this is fixed, e.g. by renaming the file. A mandatory scan and rollback on systemstart should almost disable chances for that.
(Thats where I dislike my own suggestion, because it’s generally reinventing journaling file systems in userspace)
Agree, but the IO overhead will happen by absolute sizes, so for a 40GB VM file 10% is large overhead while for an average JPEG 10% is still acceptable considering the higher complexity and risk.
Won’t be possible soon, but as I use syncthing every day (including VMs) that might put the required pressure on…
Fully agree, with no safe default the thing would be useless.
+1 from me as well:) This feature would be awesome.
It doesn’t seem like a quick enhancement but just that you guys are thinking about a possible solution is great, thumbs up.
My thoughts (forgive me my ignorance) from user’s perspective.
I/O Error Writing to Target File
What are we doing at the moment? Can’t we revert to the original “slow” sync? If “new” sync fails, (IO Error) wouldn’t be it the same for the existing protocol?
User Changed the File Locally While We Were Building The Journal
Well, I can’t think of a case where I would use the same WM image or pst file concurrently on more machines. So I’m thinking - is it possible to change file attributes to read only to prevent file mods on the file being synced? If not, maybe even rename file during this period?
Next, a few ideas about GUI. I think this should be an optional/advanced feature. Either apply to the whole folder or apply per folder + add regexp pattern like we do for ignore files. Also an option to define applicable size, i.e. >2GB, user defined value.
Currently, that’s a write error to the temporary file. Not a problem, since we haven’t touched the actual target yet. We’ll retry later and hope for better luck.
Possibly. On Windows we could use locking. But now we’re in territory where we change files in ways the user didn’t ask for (permissions) or even make the file disappear temporarily (renames). All fine since the user did ask for it by enabling in place updates, perhaps…
