I am running Syncthing between my local Unraid machine and a remote linux server. I had great success for many months until recently when sync speeds dropped to around 500kbps. I have gigabit speeds and typically get 30-50 MBps sync speeds. When I restart syncthing on either end, speeds go back to 30-50 MBps for a few minutes, before immediately dropping down to 250-500kbps. I have turned off relaying and still experience the same issue. Below is the log from the remote server from boot up to when the issue resurfaced. Nothing seems out of the ordinary:
2021-07-02 03:34:38 My ID: [remote server ID]
2021-07-02 03:34:39 Single thread SHA256 performance is 278 MB/s using crypto/sha256 (275 MB/s using minio/sha256-simd).
2021-07-02 03:34:40 Hashing performance is 234.16 MB/s
2021-07-02 03:34:40 Overall send rate is unlimited, receive rate is unlimited
2021-07-02 03:34:40 Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=[discovery id]
2021-07-02 03:34:40 Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=[discovery id]
2021-07-02 03:34:40 Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=[discovery id]
2021-07-02 03:34:40 TCP listener ([::]:22000) starting
2021-07-02 03:34:40 Ready to synchronize "completed" (76xvq-hmumr) (sendonly)
2021-07-02 03:34:40 QUIC listener ([::]:22000) starting
2021-07-02 03:34:40 ...
2021-07-02 03:34:40 GUI and API listening on [::]:8384
2021-07-02 03:34:40 Access the GUI via the following URL: http://127.0.0.1:8384/
2021-07-02 03:34:40 My name is "remote server name"
2021-07-02 03:34:40 Device [Unraid ID] is "Unraid server name" at [dynamic]
2021-07-02 03:34:44 Completed initial scan of sendonly folder "completed" (76xvq-hmumr)
2021-07-02 03:34:49 quic://0.0.0.0:22000 detected NAT type: Symmetric NAT
2021-07-02 03:34:54 Detected 1 NAT service
2021-07-02 03:35:31 Established secure connection to [Unraid ID] at 172.17.0.49:22000-73.5.10.244:24974/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
2021-07-02 03:35:31 Device [Unraid ID] client is "syncthing v1.17.0" named "f1ac3d8880d3" at 172.17.0.49:22000-73.5.10.244:24974/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
Now, the log for my Unraid server, rebooted a minute or two after the remote server:
2021-07-01 18:35:19 My ID: [Unraid ID]
2021-07-01 18:35:20 Single thread SHA256 performance is 438 MB/s using minio/sha256-simd (434 MB/s using crypto/sha256).
2021-07-01 18:35:21 Hashing performance is 376.18 MB/s
2021-07-01 18:35:21 Overall send rate is unlimited, receive rate is unlimited
2021-07-01 18:35:21 Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=[Discovery ID]
2021-07-01 18:35:21 Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=[Discovery ID]
2021-07-01 18:35:21 QUIC listener ([::]:22000) starting
2021-07-01 18:35:21 TCP listener ([::]:22000) starting
2021-07-01 18:35:21 Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=[Discovery ID]
2021-07-01 18:35:21 Using discovery mechanism: IPv4 local broadcast discovery on port 21027
2021-07-01 18:35:21 ...
2021-07-01 18:35:21 Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
2021-07-01 18:35:21 Ready to synchronize "completed" (76xvq-hmumr) (receiveonly)
2021-07-01 18:35:21 GUI and API listening on [::]:8384
2021-07-01 18:35:21 Access the GUI via the following URL: http://127.0.0.1:8384/
2021-07-01 18:35:21 My name is "f1ac3d8880d3"
2021-07-01 18:35:21 Device [remote server id] is "remote server name" at [dynamic]
2021-07-01 18:35:21 Completed initial scan of receiveonly folder "completed" (76xvq-hmumr)
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file -1"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 0"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 1"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 2"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 3"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 4"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 5"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 6"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 7"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 "completed" (76xvq-hmumr): Failed to sync 9 items
2021-07-01 18:35:21 Folder "completed" (76xvq-hmumr) isn't making sync progress - retrying in 1m0s.
2021-07-01 18:35:31 Established secure connection to [remote server ID] at [Unraid IP address]:22000-[remote server IP address]:1026/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256
2021-07-01 18:35:31 Device [remote server ID] client is "syncthing v1.17.0" named "remote server name" at [unraid IP address]:22000-[remote ip address]:1026/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256
2021-07-01 18:35:31 New NAT port mapping: external TCP address 73.5.10.244:61571 to local address [::]:22000.
2021-07-01 18:35:31 New NAT port mapping: external TCP address 73.5.10.244:24974 to local address [::]:22000.
2021-07-01 18:35:31 Detected 2 NAT services
2021-07-01 18:35:40 quic://0.0.0.0:22000 detected NAT type: Port restricted NAT
2021-07-01 18:35:40 quic://0.0.0.0:22000 resolved external address quic://73.5.10.244:22000 (via stun.syncthing.net:3478)
2021-07-01 18:35:40 Failed to exchange Hello messages with [remote server id] at [Unraid server ip address]:22000-[remote server ip address]:58672/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256: EOF
So two things seem off here. First the fact that it says no device has the required version of those 9 files. The remote server does in fact have the correct version. Second, the hello message and EOF error at the end. One other thing was that the IP addresses listed in the log of the remote server belong to neither my remote server nor my unraid server. I’m guessing that’s probably normal.
The other thing is the hello message/EOF error happened a couple minutes before the speeds fell off. Nothing was generated in either log at the precise moment when that happened.
I am not sure what I can do to troubleshoot, but any help will be greatly appreciated. I’m pulling my hair out over here. Thanks!