I was wondering what could cause the scan / sync to work in a "burst"mode.
If I check the network activity (scanning on desktop and syncing to NAS) on my NAS I get following graph.
Is this to be expected (desktop running i7 32Gb RAM, NAS is Terramaster F4-424PRO)? Seems like a sync burst +/- every minute.
I think this is generally the effect of sync preparations & finishing. When a new/changed file is announced, the pulling device first does some database work, then it prepares a temp file (including full pre-allocation of its size - if your files are large or your disks are slow, that takes a while). Only then it begins pulling data (now you see network activitiy). Finally, the data is committed to disk before a new cycle begins.
So to me this sounds like HDDs which are working in between the network transfers, probably spending most of their time preallocating empty space, flushing buffers. If you also have the database on an HDD that will also noticeably slow down database operations.
No, I have everything on cache pool (RAID1 nvme samsung 980), both data- and config location. They get moved once an hour to spin HDDs.
Then without more data I don’t know.
What more info/data would you need?
Screenshots, logs, info about the data (is it new or changed files? Small files, big files, how many? Initial scan or not?) and such. Possibly even a profile, if CPU is the suspected culprit.
I think the problem is not well defined.
Why things happening in bursts is an issue? The chart is only showing network usage. Syncthing does other things, verifies data, writes data, etc, which does not involve the network.
Probably is not an issue, but wanted to know why these burst are spread in time. When I look at the status now, same sync running on same size files, I get a complete different view.
So just want to understand why the difference…
Sync speed is considerably higher now than with the first post.
I don’t think anyone can answer anything by looking at a single chart.
There is simply not enough context to answer this question.
Syncthing has a lot of optimisations to try and avoid hitting the network, i.e., if the changed file is very similar to the previous version of the file, it will only transfer blocks that have changed, and copy same blocks from previous file. If there are other files in the folder already that have the same blocks, we’ll do a copy instead of a transfer, etc.
Perhaps you are hitting these optimisations, but there is just not enough details to give you a real answer. All we have is a single picture of a single metric.
Are you running Syncthing on both devices or are you mounting a network share?
How much data in total are we talking about? There’s quite a difference between saturating a link for 0.5s vs 10s.
IMHO it’s strange that the traffic is 100% inbound. Are you certain that this is caused by Syncthing?
Running syncthing on both devices, no mapping or network share.
Nothing else running on desktop and syncthing is only docker app installed on nas. Mover on unraid running once every hour, cache doesn’t get above 60% or is being moved to hdd.
We are talking about 1.2 Tb, 30k files of approx 40meg each.
