slowing down over time

Hi all,

currently i have 2 unraid servers with each several disks and i would like to duplicate my collection of large files, around 70TB in total. Both nodes have 10GBe cards installed and properly configured with a direct connection without switch inbetween. Node 1 is sendonly, Node 2 is receiveonly.

The sync is starting quickly with 150-250 MB/s via LAN which is the single disk speed, all good. After around 2 days it slows down to below 20 MB/s… then i click in syncthing UI on restart in sender and receiver - and it jumps back to >200MB/s. I did this procedure now 3-4 times in the last week and i can reproduce it.

Is there anything i can log to check why this happens ? As said, without any reboot, no docker restart, no container restart … just syncthing restart fixes it back to max speed.

1 Like

A support bundle taken when things are fast and another when they are slow might be helpful.

support-bundle-VZ5JV5A-2024-11-04T005556.zip (90.1 KB) support-bundle-VZ5JV5A-2024-11-04T014811.zip (84.4 KB) syncthing-heap-linux-amd64-v1.28.0-005616.pprof (51.8 KB) syncthing-cpu-linux-amd64-v1.28.0-005632.pprof (27.8 KB)

and here the next - after i restarted the receiver after absolutely nothing was copied any more

support-bundle-VZ5JV5A-2024-11-04T015641.zip (72.5 KB)

right after synthing restart, back to 130MB/s

And again … new logfiles after it “stopped” … and after restart support-bundle-VZ5JV5A-2024-11-04T021755.zip (67.5 KB) support-bundle-VZ5JV5A-2024-11-04T021531.zip (75.1 KB)

I’m not finding anything specific here, unfortunately. What could potentially shed some light are goroutine and block profiles, which are not included in the support bundle. To grab those, run the problematic side with the full profiler enabled, syncthing serve --debug-profiler-listen=127.0.0.1:9093, and when it’s slow grab profiles:

curl -o goroutine.pprof http://localhost:9093/debug/pprof/goroutine
curl -o block.pprof http://localhost:9093/debug/pprof/block
1 Like