my syncthing setup is very slow (tens of kilobytes per second, with few instant mbit burst up and there) and I’ve no idea how to debug it.
So far:
gbe switched network, 2 hosts only (freenas and windows).
CPU/RAM usages are very low. tested.
Disks on both sides (raidz1 on wd-reds, and ntfs on nvme) are way faster. tested.
scp between those 2 machines have about 250mbit/s throughput. tested.
I replaced SyncTrayzor with console syncthing.exe distribution.
I downgraded the windows side app to match the one on freenas (they are both 0.14.51 now), deleted the index folder and waited a bit to let it rehash the whole directory. no changes.
Thanks Jakob but unfortunately that’s not the problem: both hosts report the local rfc1918 addresses.
And the CPU is fine; there are 8 2.4GHz cores on that machine, and the overall usage is mostly under 5%. Just for reference:
06:42:18 INFO: Single thread SHA256 performance is 255 MB/s using minio/sha256-simd (218 MB/s using crypto/sha256).
06:42:18 INFO: Hashing performance is 203.79 MB/s.
In any case, I tried to rise the clock back to the stock frequency (3.5GHz) and didn’t change anything; I keep the cpu underclocked in order to minimize heat and noise.
In the while I noticed
06:42:18 INFO: Overall send rate is unlimited, receive rate is unlimited
06:42:18 INFO: Rate limits do not apply to LAN connections
06:42:18 INFO: TCP listener ([::]:22000) starting
06:42:18 INFO: GUI and API listening on 127.0.0.1:8384
So I’m currently searching for a way to completely disable ipv6 on Freenas; it happens sometimes that a not properly ipv6 network can throw in erratic behavior with some apps. Being distributed/installed as an app the system is immutable apart for a few configuration files, so I can’t follow FreeBSD generic advice.
EDIT: after rebooting the freenas machine, the single thread performance halved and the tcp listener is correctly bound to an ipv4 local address. And I can’t access the interface any more; nginx (the front end proxy) gets a ‘connection refused’ upstream (127.0.0.1:8384); but the syncthing daemon is working in background, I can see it connecting to the window machine and (slow) sync’ing.
Did you check the actual load on the freenas machine, especially disk access or iowait? I saw that you wrote that disk shouldn’t be a bottleneck, but it often still is on spinning disk due to seeking.
I tried to upload a screenshot of daily load graph, but it disappear after the upload to this board. I’ve no idea how to attach it to this message.
The system is not doing anything yet; it’s a fresh install. I’ve no idea how to check for iowaits on this ‘rom version’ of freebsd as I can’t use the package system.
The disks are a raidz1 pool (zfs); I’m not 100% comfortable with ZFS, but any disk activity I tried went smooth and didn’t clog the system. I should spend some time to refresh my memories about ZFS, as sometimes there are small details that can degrade performance for some use cases (ex: COW is not good for storing big VM images) and not others. Thinking about it, I should try to disable atime, as an example… dd’ing devzero to file doesn’t rely on atime as syncthing might do…
@Audrius
I shared the windows home folder only. Local syncthing reports 118.176 files, 11.904 folders, 39.5GB. I must change workload to something simpler, but I don’t have any suitable subfolder atm. I saw the scanning process reporting failed file scans but they are mostly the browser cache or similar temporarly locked files; is it a problem for syncthing?