Master Node
Windows 10 PC, i5, 8Gb, 64bit.
Seagate 2TB, 64MB cache, 5900 rpm, 6Gb/s.
Virtual memory automatically set 4098MB.
Normal Node
Windows 10 PC, i3, 2Gb, 64bit (Runs Syncthing only).
WD 1TB, 64MB cache, 7200 rpm, 6Gb/s (sorry not the same as above).
Virtual memory automatically set 1152MB.
Other
Connected over LAN.
Both nodes set to Full Compression.
Shared Folder 185GB, 371 Files, 120 Folders (most files are 4GB).
Observations
When I make a small change to a text file, the scanning process on the Master node takes approx. 15 seconds. The Normal node then takes approx 330 seconds to scan (22x longer).
None technical question
I was wondering what is the reason for the difference? I understand the hardware is different on the Normal node, but it is still a reasonable PC. During the scanning process memory usage peaked at 60% and Disk usage peaked at 50%.
Would it be worth adding more memory in the Normal node to see if this makes a difference?
Would it be worth matching hard drives?
Does Full Compression make a difference on the Normal node?
You shouldn’t use full compression (especially when both are in the same LAN). Compression adds significant CPU usage with not much less data to transfer (unless you have 185GB text files, which can be compressed easily). But that setting doesn’t impact scanning performance, only transfer speed and resource usage while transferring.
The text file must be really big, if it takes so long to scan. How big is the text file that you change?
Did you append text at the end of the file or did you change something in the middle?
Please check the logs and compare the stated single core hash performance.
Syncthing doesn’t do anything different when scanning regarding the type of folder.
The txt file itself is 55 bytes and I just add a line at the end to force a change.
I did not mention that Syncthing is not synching the folders & file directly.
I use Syncthing to Mirror the ‘local’ output files of 3rd party backup software. The files created by the 3rd party software are encrypted and as described “Shared Folder 185GB, 371 Files, 120 Folders (most files are 4GB in size)”. At worst the only thing to have changed would have been a very small part of one the 4GB files, though this will take some investigation to confirm.
I will need to learn how to “Check the logs and compare the stated single core hash performance” but will hopefully be able to reply later today.
Performance should be down to the hardware, in this case. The “normal” node may need to page data to/from virtual memory more often since it has significantly less physical memory, and the i3 processor is likely not as fast at calculating hashes as the master node’s processor.
Still, 15 seconds to scan a text file is, by itself, probably on the long side…
Depending on how you use / start Syncthing, you have several options to get to the log.
If you use Synctrayzor, you have the log at the bottom of the window. If you cannot scroll back to the start of syncthing, you can restart syncthing to get a fresh log.
If you use syncthing “bare bones” without any wrapper App, you need to start Syncthing in a “Dos Box” to see the output.
The hash performance is one of the first lines from Syncthings output.
Syncthing splits the files in 128 Kbyte blocks. If your updated 4GB backup file has a few bytes more in the middle, it will have to scan (and transfer) the complete file after that. So it has much more to do than just get the few new bytes.
Also, as the backup files are encrypted (and probably also compressed) the complete file could be different, not just a few new bytes in the middle.
Scanning a text file should not take that long, unless it’s huge. The rest is attributed to disk speed and more likely cpu, simple as that, nothing todo with the mode. I bet the speed numbers printed as you start syncthing will reflect the difference.
The stated single core hash performance on the Master is 155 MB/s.
The stated single core hash performance on the Normal is 182 MB/s, which surprised me.
I swapped the memory over to test. The Master is currently fitted with 2GB & the Normal fitted with 8GB. The Normal scan now only takes 38 seconds to complete, which is much better.
I would not normally attempt to run Windows 10 with only 2GB! I must have taken some memory out of the test PC in the past and forgotten.
The difference in hash performance is probably, because the i3 is already set to maximum speed while the i7 is in low speed / powersafe (because it doesn’t have much to do) and it doesn’t switch to full speed fast enough to have the hash speed test of Syncthing running with full speed.
The hash performance on my home server varies between 94 MB/s and 345 MB/s, while mostly being between 180 and 200 MB/s. This all depends how much the CPU has to do at the time and how the CPU clock speed is set.
Thanks, this is all good to know. After upgrade v0.14.7, I have seen hash performance reported between 220 and 271. I have also ordered 8Gb of memory for my Normal node.