I just mass GPS tagged thousands of raw images, each 20 MiB in size. This was a while ago. I realize that Syncthing is still downloading:
309 GiB for 15,250 items is: 20 MiB / file
So it looks like it’s downloading every single raw image again.
Shouldn’t it only sync the meta data?
The filesystem of the peer is Arch Linux with NILFS2.
GPS tagging is, from a pure image perspective, a meta data change. But from a file perspective it is not. GPS tagging modifies the file, more specifically the EXIF section in the file.
EXIF is metadata. I expect Syncthing to only sync the blocks that changed. So it should’t matter if the metadata is stored inside or outside the file.
I wonder if that is an issue related to my use of NILFS2. Or maybe Syncthing doesn’t actually download the whole files, even if that shows in the web UI.
I clarified my post above. EXIF is, from a file perspective, NOT metadata.
I don’t know the details about modified blocks.
Out of Sync Items refers to the number and size of the files that are out of sync, not how much data there is to download.
When you change the files (as stated previously, changing the metadata for the raw images changes the data of the file) this is expected behavior for Syncthing.
What I don’t know yet is how many blocks and how much data will need to be transmitted for the changed files to sync.
I see how that can be confusing. Should clarify the topic, but I cannot edit it anymore.
Anyhow, my question stands.
Is there a way to see how much data is actually transmitted?
The addition of a GPS location to EXIF should be really small, probably less than a kilobyte of data changed.
I don’t know enough about NILFS2 to say if that impacts anything – my belief has been that Syncthing is agnostic as to filesystem, but that doesn’t mean that the filesystem can’t impact performance or other factors.
Syncthing divides files into blocks and transmits changed blocks. Syncthing can re-use blocks from other files (if you have 1000 identical files to sync, Syncthing should be sending only blocks for the first file synced and then rebuild the others from there). This means that you can hope that the changes you made affected few blocks.
Upload Rate and Download Rate are shown under This Device and each Remote Device, including a total. I believe the total is shown for This Device since Syncthing started, and for each Remote Device since the connection between Syncthing instances was established.
From the rate, it is hard to tell. Is there no additional information? A log?
I really would really like to see what is going on. This is a huge amount of data.
Details on how synchronization works are here: Understanding Synchronization — Syncthing documentation .
The log is found in the GUI under Actions / Logs. Debugging facilities are in a tab found in that page.
There’s no way I know of to figure out how many blocks are changed and need to be transmitted over the network except to let Syncthing do the work and let you know.
If you wanted to test this for measurement purposes, I would probably do this:
There is a counter for data sent/received on the device tab, though.
While files are syncing, you can see on the receiving side how much is left to sync. Clicking on those numbers opens a dialog that shows which files are currently being worked on. It contains a color coding to tell you how much data of the currently active files has been downloaded already, how much is re-used from other files, and so on. But that is only on the receiving side, where the folder is still in “Syncing” state.
Right. I’ve looked at that dialog many times but never paid attention to the color codes. Here’s a recording:
Observations:
Only once during the recording, the bar is filled all blue except for one red stripe.
Generally there is only one red stripe. The rest is gray. I’m not sure what this means. Maybe the syncing per file happens too fast in order to fill the rest with blue.
Indeed, the syncing seems to be fast. But, according to speedtest.net, the download rate is 244.58 Mbps, or 30 mega bytes per second. At that speed, it could download three 20 mega byte raw files files every two seconds.
Under the assumption that it indeed only downloads a tiny delta for each file, I wonder why it’s not faster. There is quite a delay after each two files.
OK, I just restarted. That counter in the tab This Device is currently at 237 KiB for downloads. The folder in question shows ~169 GiB out of sync items. I’ll let it run through the night, then check back.
Syncthing is very much not capped by the raw download speed. It does a lot of checks to make sure your data is safe and sound. It records block hashes in a database and verifies those blocks against the hashes. Time spent comparing or copying already existing blocks locally is time not spent with network transfers.
Trust how it works and be patient. Syncthing tries to minimize network transfers which can be avoided by copying locally. It’s hard to see exactly how it does that, because the GUI update interval is probably too slow. But it is happening.
Also note that Syncthing is not optimized for raw transfer speed, but rather data safety and management to possibly keep a whole cluster of devices all synchronized. It’s made to run continuously in the background and stuff just being where you expect it, even if not at the fastest possible transfer rate.
Thanks, that sounds sensible. Well, I’ll see tomorrow how much data has actually been synced for the 169 GiB.
Thanks for pointing me to this excellent article! It makes it quite clear that the underlying file system has no influence on what is happening. (except regarding case-sensitivity, which this thread is not about)
Cool, I never realized.
Thanks. This is somewhat what I’m doing now, at least the looking at the stats part, in the This Device tab.
Summary of my observation:
8,245 items totaling 169 GiB were downloaded, plus some more items that I didn’t track. To get more reliable data, I should’ve stopped syncing of other folders.
Most of these items were 20 MiB raw image files.
At the start, in the evening, the download counter under This Device was 237 KiB.
Once finished, in the morning, the download counter under This Device was 5.08 GiB.
Conclusion: Per 20 MiB raw image file, no more than 646 KiB were uploaded. So all good, my worries were unfounded. SyncThing impressed again!
