Extremely slow transfer speed fixed by restarting, resurfaces after a few minutes

I am running Syncthing between my local Unraid machine and a remote linux server. I had great success for many months until recently when sync speeds dropped to around 500kbps. I have gigabit speeds and typically get 30-50 MBps sync speeds. When I restart syncthing on either end, speeds go back to 30-50 MBps for a few minutes, before immediately dropping down to 250-500kbps. I have turned off relaying and still experience the same issue. Below is the log from the remote server from boot up to when the issue resurfaced. Nothing seems out of the ordinary:

2021-07-02 03:34:38 My ID: [remote server ID]
2021-07-02 03:34:39 Single thread SHA256 performance is 278 MB/s using crypto/sha256 (275 MB/s using minio/sha256-simd).
2021-07-02 03:34:40 Hashing performance is 234.16 MB/s
2021-07-02 03:34:40 Overall send rate is unlimited, receive rate is unlimited
2021-07-02 03:34:40 Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=[discovery id]
2021-07-02 03:34:40 Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=[discovery id]
2021-07-02 03:34:40 Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=[discovery id]
2021-07-02 03:34:40 TCP listener ([::]:22000) starting
2021-07-02 03:34:40 Ready to synchronize "completed" (76xvq-hmumr) (sendonly)
2021-07-02 03:34:40 QUIC listener ([::]:22000) starting
2021-07-02 03:34:40 ...
2021-07-02 03:34:40 GUI and API listening on [::]:8384
2021-07-02 03:34:40 Access the GUI via the following URL: http://127.0.0.1:8384/
2021-07-02 03:34:40 My name is "remote server name"
2021-07-02 03:34:40 Device [Unraid ID] is "Unraid server name" at [dynamic]
2021-07-02 03:34:44 Completed initial scan of sendonly folder "completed" (76xvq-hmumr)
2021-07-02 03:34:49 quic://0.0.0.0:22000 detected NAT type: Symmetric NAT
2021-07-02 03:34:54 Detected 1 NAT service
2021-07-02 03:35:31 Established secure connection to [Unraid ID] at 172.17.0.49:22000-73.5.10.244:24974/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
2021-07-02 03:35:31 Device [Unraid ID] client is "syncthing v1.17.0" named "f1ac3d8880d3" at 172.17.0.49:22000-73.5.10.244:24974/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256

Now, the log for my Unraid server, rebooted a minute or two after the remote server:

2021-07-01 18:35:19 My ID: [Unraid ID]
2021-07-01 18:35:20 Single thread SHA256 performance is 438 MB/s using minio/sha256-simd (434 MB/s using crypto/sha256).
2021-07-01 18:35:21 Hashing performance is 376.18 MB/s
2021-07-01 18:35:21 Overall send rate is unlimited, receive rate is unlimited
2021-07-01 18:35:21 Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=[Discovery ID]
2021-07-01 18:35:21 Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=[Discovery ID]
2021-07-01 18:35:21 QUIC listener ([::]:22000) starting
2021-07-01 18:35:21 TCP listener ([::]:22000) starting
2021-07-01 18:35:21 Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=[Discovery ID]
2021-07-01 18:35:21 Using discovery mechanism: IPv4 local broadcast discovery on port 21027
2021-07-01 18:35:21 ...
2021-07-01 18:35:21 Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
2021-07-01 18:35:21 Ready to synchronize "completed" (76xvq-hmumr) (receiveonly)
2021-07-01 18:35:21 GUI and API listening on [::]:8384
2021-07-01 18:35:21 Access the GUI via the following URL: http://127.0.0.1:8384/
2021-07-01 18:35:21 My name is "f1ac3d8880d3"
2021-07-01 18:35:21 Device [remote server id] is "remote server name" at [dynamic]
2021-07-01 18:35:21 Completed initial scan of receiveonly folder "completed" (76xvq-hmumr)
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file -1"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 0"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 1"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 2"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 3"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 4"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 5"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 6"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 Puller (folder "completed" (76xvq-hmumr), item "file 7"): syncing: no connected device has the required version of this file
2021-07-01 18:35:21 "completed" (76xvq-hmumr): Failed to sync 9 items
2021-07-01 18:35:21 Folder "completed" (76xvq-hmumr) isn't making sync progress - retrying in 1m0s.
2021-07-01 18:35:31 Established secure connection to [remote server ID] at [Unraid IP address]:22000-[remote server IP address]:1026/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256
2021-07-01 18:35:31 Device [remote server ID] client is "syncthing v1.17.0" named "remote server name" at [unraid IP address]:22000-[remote ip address]:1026/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256
2021-07-01 18:35:31 New NAT port mapping: external TCP address 73.5.10.244:61571 to local address [::]:22000.
2021-07-01 18:35:31 New NAT port mapping: external TCP address 73.5.10.244:24974 to local address [::]:22000.
2021-07-01 18:35:31 Detected 2 NAT services
2021-07-01 18:35:40 quic://0.0.0.0:22000 detected NAT type: Port restricted NAT
2021-07-01 18:35:40 quic://0.0.0.0:22000 resolved external address quic://73.5.10.244:22000 (via stun.syncthing.net:3478)
2021-07-01 18:35:40 Failed to exchange Hello messages with [remote server id] at [Unraid server ip address]:22000-[remote server ip address]:58672/tcp-server/TLS1.3-TLS_AES_128_GCM_SHA256: EOF

So two things seem off here. First the fact that it says no device has the required version of those 9 files. The remote server does in fact have the correct version. Second, the hello message and EOF error at the end. One other thing was that the IP addresses listed in the log of the remote server belong to neither my remote server nor my unraid server. I’m guessing that’s probably normal.

The other thing is the hello message/EOF error happened a couple minutes before the speeds fell off. Nothing was generated in either log at the precise moment when that happened.

I am not sure what I can do to troubleshoot, but any help will be greatly appreciated. I’m pulling my hair out over here. Thanks!

In the ui you can see the address the device is connected on. Suggest you check if there is a difference in protocol used/addresses when its slow vs when its fast, on both devices.

There’s not a difference on either the client or server side. I also did a clean install on both ends and it fixed the issue for maybe an hour or two, but it came back.

Sorry, but there isn’t much I can suggest.

I guess try running iperf for a long period of time and see if you observe something like that.

Transfer speeds are heavily influenced by what you are transfering. Namely the per file overhead can make transfer speeds quite bad, so I guess make sure you are seeing the same with a single large file.

Which addresses?

Also, depending on what changes you’re syncing, the bottleneck might be copying local data and not transferring over the network.

The connection addresses both in the log and GUI on the remote server side. This is the log from the remote side:

2021-07-05 14:58:32 Device [Unraid ID] is "Unraid name" at [dynamic]
2021-07-05 14:58:32 Established secure connection to [Unraid ID] at [server address]:22000-[local address]:61571/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256

The second line doesn’t reflect the same addresses in the log on my undraid server. The local address correctly corresponds to my home IP address. The log on my server side shows the GUI IP address. The remote server address is also different than what the server address is listed as on my unraid server. Is this normal? How could I go about setting the addresses to be fixed and not dynamic? Perhaps that could help the issue.

Also, this issue happens regardless of the files I am syncing. Most of the files I sync are large ~30GB files and I have had no issue historically with speeds. The speed drops off in the middle of the files, not when it’s moving to new files.

If the devices are on different networks, and you are behind a NAT (network address translator), then it’s expected that the local address will not match the address you see on the other side.

Namely, my local machine ip address is 192.168.0.10 in my local network, but my IP on the internet is completely different and assigned by my ISP, so when I connect to things on the internet, everyone see’s the IP my ISP gave me, not 192.168.0.10.

How to configured addresses is covered in the docs.

Would I want to change the listen address? It lists various options in the documentation but not sure which would be recommended.

Not sure I can’t advise you, as I still don’t understand what the underlying problem is, and whether the ips mismatch because you are behind a NAT or not.

yes, IP mismatch was because of the NAT

So I am not sure what your action plan is?

As I suggested above, the fact the IPs mismatching might be expected, if like I assumed, you are talking about two devices behind different NATs.

You’ve haven’t given enough information about the network topology, the IPs you would expect to see (are they public ips or local ips) etc, for me to give you any meaningful advice.

I’m sorry. As you may have guessed, I’m not very experienced with networking stuff. Disregard the IP address stuff - I was just confused by NAT. Please let me know what additional information would be helpful to troubleshoot the issue.

I attempted to set this up on my Windows machine as well to see if the issue persisted. I am getting perpetual “preparing to sync” on the windows end and the “failed to exchange hello messages EOF” error. Not sure what I can do to troubleshoot the EOF error which seems to be common between the two instances of syncthing in my home

Could you set up iperf3 on both nodes and try a long running test? It would be good if we could rule out any network level problems.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.