Hi all,
Is it possible to limit the hashing speed when scanning new files? We use syncthing to sync media between Avid Media Composer suites using QNAP devices but scanning at full rate can hurt performance via SMB/AFP (the QNAP can hash at 300+MB/sec which can exhaust the RAID).
I see theres a hashers option but even with just 1 this seems to still overwhelm. Is there any way to limit this further?
Thanks so much, Jon
You can set the environment variable GOMAXPROCS=1 when starting Syncthing, so that only one CPU core will be used. If you’ve got multiple folders, this means that only one folder will be processed at any given time. With this variable set, you also don’t need to touch hashers or any other CPU-bound settings, as with only one core available, they all will default to 1.
Now, if limiting Syncthing to only one CPU core is still not enough, then you probably need to look into the OS and possible solutions to limit the CPU even more. One way to do it is to run Syncthing with a lower CPU and/or I/O priority. QNAP NAS run Linux, so you may look up nice and/or ionice, etc.
Alternatively, you could simply underclock the CPU. This should even be possible from within the operating system itself.
What’s the point of playing around with GOMAXPROCS when numbers of hashers is already set to 1, and the issue is with disk IO not cpu?
Syncthing already lowers it’s priority. I’d first check that really works ionice -p <syncthings-pid>.
And where is your syncthing DB located? If thats on the raid. move it to something else (ideally SSD, if you have enough RAM that’s an option too, or at least a non-raid disk).
Because hashers is per-folder, so there may be multiple folders being scanned at the same time, each utilising one CPU core. With GOMAXPROCS, Syncthing is limited to using 1 core/thread period.
I didn’t think it was really clear from the explanation whether the problem was in CPU or the disk. If the CPU is super-slow (or artificially limited to such a state), hashing will be slow even when scanning files on a very fast storage like a RAM disk 
.
Right, so they also need to set maxFolderConcurrency to 1. Still I don’t like fiddling around with low-level switches like that, it will potentially have other undesired effects.
And yeah sure, actual data about what’s the bottle neck on the OPs systems would indeed be nice - there’s a good amount of guessing done here 
Yeah, that’s probably more preferable 
. I’ve just found GOMAXPROCS a more convenient/lazy solution as a single switch to deal with performance problems on low-end hardware 
.
I’ve re-read the original question once again. In my understanding, the RAID storage is located on a different device and shared through SMB with the QNAP NAS. The problem is about the NAS boggling down the RAID storage on the remote device when performing scanning.
If that’s the case, limiting priority/niceness on the NAS locally won’t really help, will it? Would limiting network bandwidth have possibly any effect here?
Of course, more information and concrete numbers on whether the “300+MB/sec” hashing speed depends more on the CPU or is purely storage-bound would also be welcome.
I’ll wait for the OP to clarify what the setup is.
Hi all,
Thanks for all the replies and apologies the original post wasn’t particularly clear.
We use syncthing on two device types, a 4-bay unit that doesn’t have AES-NI and a 6-bay unit that does.
Both devices run a custom debian build booted from factory DOM. Both devices need to be fully encrypted, we were using fscrypt but have now moved to LUKS on BTRFS (to take advantage of st’s CoW support). The smaller unit uses Adiantum due to the lack of AES-NI.
Sequential access to the RAIDs is roughly: 350MB/sec for the 6-bay 200MB/sec for the 4-bay
On the 4 bay unit, hashing/scanning doesn’t affect the RAID much because the CPU is the limitation (st clocks its single thread hash speed at 65MB/sec). This is perfect, as it leaves plenty of overhead for access.
On the 6 bay unit, st clocks hash speed at 500+MB/sec… therefore when scanning it can exhaust the entire RAID leaving little room for access to it via SMB/AFP (which is running locally on the box).
The question really is whether Its possible to set a particular max speed for hashing without resorting to slowing the whole app down by limiting it at OS level.
– Just to add, we already use GOMAXPROCS to limit two 2 cores (and therefore a concurrency of 2) and are running two st nodes on each box (1 to sync media and one to priority sync projects).
Thanks Jon
It is not, sorry.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.