Scanning and Syncing on low end hardware causes major overall slowdown

Hi Guys,

i wasn’t sure if i should open up this as a feature request or just a simple support thread as there may be a solution to this already.

Today i discovered that Syncthing runs extremely slow on my ds213j synology NAS as a result of both scanning a huge folder while also not pausing syncing. So i added about 700GB of large files to my folder (r/w in both directions) and syncthing started scanning these files. It did scan those files for as slowly as 500kB - 1 MB/sec resulting in an approximate time remaining of 3 days or longer. Yet as soon as i paused the sync with my remote folder, the process sped up to about 14 MB/sec having scanned those 700GB in about 4 hrs. And as for syncing, speeds were below 1 MB/sec while scanning, so no bargain there too.

CPU utilisation was in both cases 100% and RAM usage did not change a lot. There were no other activities on the NAS while testing. Even though my HD speeds are up to 90MB/sec read, it is most likely that the CPU is bottlenecking when simultaniously scanning and syncing at the same time.

Now my proposal would be to get an option to auto-pause all syncs while folders are being scanned for low end devices.

What do you think?

Syncthing uses encryption and cryptographic hashing to do it’s job.

The device you have it on is fairly weak for the purpose you are using it for, hence why you have these problems.

Sadly, weak devices + a lot of data is not the target market for syncthing, especially when syncing terabytes, so I don’t think we’ll be adding specific hacks for devices that are being abused.

i wouldn’t say i am abusing my system… It’s been working great for my bandwidths and folders which are <100GB, even though my system is on the lower end.

There are other things you could try. Moving syncthing’s database to a separate directory, getting faster drives.

I suspect the 100% CPU usage is most likely because of slow IO.

This. Spinning disks don’t particularly like doing many things in parallel. Scanning and syncing different folders on the same disk at the same time will make both slower.

But of course also the CPU stuff.

Why not the other way around? What’s a low end device? :slight_smile:

actually, i already tested other approaches

Setup 1 was: syncing a folder from another disc (not where syncthing is installed) with a remote location

Setup 2 was: syncing a folder from the same disc where syncthing is installed

Setup 3, as in: syncfolder is in the same directory as the database was never the case.

So i actually changed to setup 2 because i found out, setup 1 was slower. This might be because (and this is my assumption) syncthing actually grabs parts of the files and copies it to its home folder and then scans or syncs it. Because whenever i was syncing and scanning, both my discs had the same read and write speeds at the same time, while i thought it would only need the one disc where the data was stored and RAM and CPU for the rest of the work. But it seems as if syncthing is caching all the data in its own homefolder. Thats at least my observation.

Setup 2 is better in terms of performance and Setup 3 was never used. I never used the default directories and setup my own.

Why not the other way around? What’s a low end device? :slight_smile:

Why not the other way around: Might be dependent on for what you use those shares of course. For me, there is nothing to transfer if the folders don’t get scanned prior, thats why i suggested it.

Low end devices aren’t actually extremely uncommon, but just rare. Your own statistic provides the data for that. About 5% of all users do have similar computing power as to my setup. Yet i have to admit, they handle way less data than i am.

But that brings me back to my second answer. Other than this hiccup, syncthing works perfect even though my low end hardware. Having an opt-in to a more conditionalised approach in terms of “what job to do first” would be nice in my case.

I’m not suggesting low end devices don’t exist, I’m sure they are rather common, just wondering what criteria you’re considering. In terms of scanning+syncing folders on the same physical disk, anything with rotating disks is low end and will behave much the same I think.

CPU wise I don’t think low end vs high end matters much - doing A plus B concurrently will in total take roughly as long as doing first A then B. If there are lots of cores then doing the tasks concurrently will be faster, but even with just the one core the difference isn’t night and day and it’s not something I think it’s worth optimizing for. (We already de-optimize CPU performance on Windows and Mac to avoid hurting system interactivity too much.)

Disks have the other property, that doing A and B concurrently can take ten times as long as doing first A then B.

I am not suggesting you bring this upon certain users which low end devices. I am merely suggesting to think about an opt-in in the advanced folder section menu. Helppage would vaguely say that this might improve performance on low and devices such as embedded devices with lots of data to handle.

I am not sure if you want to tell me to let it be or if my problem is not clear enough. I’m fine either way as i at least tried :slight_smile: If you think there is definitively no room for that, it’s ok for me. Maybe i’ll stick with it, maybe i’ll check if rclone performes better on my embedded device. I merely wanted to bring up the issue and suggest a work around for it.

Also, these scans are one off, so you can do the “optimisation” manually by hand once, and then you should be over your hump.

Having code as workarounds for one-off scenarios of abuse makes me sad.

We used to have a mandatory scan on startup (which prevents syncing), which I think should sort of already address this.

Also, when I say abuse I don’t mean that you are doing something wrong.

I just think that expectations should be adjusted to match the ratio of hardware to data being handled, and accept that manual fiddling might be necessary at this point.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.