connection speed high and then drops and erratic

Support
1

Hi all. I’m hoping someone will recognize these symptoms. I have two high end Qnap NAS’s connecting over the internet on 1gbit connection. I did the initial sync of these nas with syncthing on a local 10 GigE network and that worked at around 300 MB per second. When I connect them over the internet they start for the first minute or so at 180 Mbps and are nice and constant. Then the speed drops down to an average 8 Mbps and it becomes spiky where I get speeds of around 80 Mbps for a second and the right down to zero for a second or two and back again like a pulse. If I pause the sync and start it up again I once again get a short period of high consistent speed and then it drops back to that pulsing speed. This interaction only happens in one direction. data flow from NAS A to B is smooth and consistent (although at relatively low speeds Due to asynchronous internet connection on A). From NAS B to A however I get this interaction.

Both locations have port 22000 forwarded to the qnaps. Both sides have the other sides external facing IP address and port 22000 entered instead of dynamic for an address and I have turned off relays.

Does anyone have thoughts on what I am missing?

2

Possibly that it’s syncing changes where most of the data can be reused locally, so you’re seeing mostly disk I/O without network traffic, punctuated by bursts of network traffic for missing blocks.

3

Thanks for the response. I just checked and if I look at the out of sync items that it is working on they are big files (3 to 4 gig each) with nothing local to reuse. Here is a screen cap of my traffic on the router (about 45 seconds worth). Blue is upload and orange is download. Download should easily sustain 30 to 40 MB per second and does often when I first initiate the connection but for less then a minute. Then its that erratic mess.

Capture
Capture748×533 46.1 KB

4

But everything seems to work if you’re using LAN connections. Which router are you using? What is the latency of the connection from NAS A to B?

5

On this end (nas A) is behind an Asus AC68U with Merlin firmware, NAS B is behind a HH3000 modem/router. Latency is good as far as I can see at around 40ms. Is there a way to test or see the latency specifically of syncthings traffic between nodes?

As a test I set up an RTRR sync between the two nas and I get sustained 220 to 300 Mbit transfer with that so about 10 times syncthing over the same network. I ran that transfer for 800 gigabytes of data with no issues. I followed the same process of forwarding the correct ports on either end to the NAS RTRR servers. So it seems like nothing in the routers is slowing things down which in my mind leaves me with something specific to syncthing and its setup.

6

Did you look at disk I/O on the destination during the transfer? (And CPU/RAM utilization – that is, normal “it’s slower than I expect, wonder what’s going on” investigation.)

1 Like
7

Both are 8 disk raids capable of 1TB per second, Disk access is in single percents and cpu is around 15%, Ram utilization at 20% or less

8

Then I’m not sure. I suggest exploring the various trace options and perhaps profiling, if there’s no actual resource that seems to be bottlenecking.

9

Is there a way to specifically traceroute how Syncthing is connecting? I feel any traceroute I do at this point isn’t valid since it doesn’t necessarily mirror whatever syncthing is doing to make the connection.

10

You can see in the GUI or logs what protocol and address it’s connecting via. (If it’s not a direct TCP connection that’s your first problem, but then I don’t think you’d see the speeds you are seeing either.)

11

Yep the GUI and log only mentions a direct TCP connection to the other machine. No other addresses are mentioned. I did a bit more fiddling and now have turned off all discovery and announce as well as nat in the advance options. I manually entered addresses and ports for the devices. Up side is now for whatever reason I get a consistent speed but it is now a mere 2-4 Mbps.

I am at a complete loss at this point

12

At this point i’d strongly suggest capturing a dump of the network communication.

13

I have the same issue. Between two unraid server with docker Syncthing.

Usually can transfert at 50-60MiB/s and 92MiB/s with two folders at same times.

Now with the latest version of syncthing on both servers, the speed is correct (50-60MiB/s) but 1min after drop to 1MiB/s with spikes 2.5MiB/s

My disks are not fully busy

14

Does QNAP offer tcpdump?

15

It doesn’t but I now have it installed. Unfortunately I have reached the edge of my experience in networking. I’m not sure what I am looking for with tcpdump so if you have set of options I should run the command with and an idea of what I might be looking for that would be great, I’ll give it a whirl.

16

My first reaction to this type of network activity is to blame the ISP on one of the sides. I have two nodes that exbibit this behavior and discovered that my ISP on the receiving side was resting connections; they refer to this as “traffic management”. When any particular connection takes up a significant portion of the download bandwidth the ISP sends a reset causing retransmissions and a reestablishment of the connection. This causes the bandwidth graph ramp up and then when the reset happens it drops back to 0 and ramps back up until the ISP resets it again. I was able to resolve this by creating a VPN tunnel from one node to the other. In my case the ISP does not “traffic manage” vpn connections.

1 Like
17

https://www.wireshark.org/docs/wsug_html_chunked/AppToolstcpdump.html

e.g:

tcpdump -i eno0 -s 65535 -w syncthing.dump -q tcp port 22000

You might need to change eno0 to the network interface of your device. Once you have captured enough traffic hit Ctrl+c to stop tcpdump.

Just capture a chunk of the traffic and upload the dump file somewhere so we can take a look.

18

here is a dump using those settings. During this dump I was getting around 1 to 2 MB per second Vs the 30MB/second I get with RTRR sync transfers.

19

According to Wireshark about 25% of your packets are a lot bigger than the usual MTU of ethernet connections. Did you configure Jumbo Frames on your NAS?

==================================================================================================================================
Packet Lengths:
Topic / Item       Count         Average       Min Val       Max Val       Rate (ms)     Percent       Burst Rate    Burst Start  
----------------------------------------------------------------------------------------------------------------------------------
Packet Lengths     2293          1498,34       66            9257          0,5834        100%          1,1200        2,151        
 0-19              0             -             -             -             0,0000        0,00%         -             -            
 20-39             0             -             -             -             0,0000        0,00%         -             -            
 40-79             825           67,05         66            78            0,2099        35,98%        0,4000        3,595        
 80-159            82            92,73         86            94            0,0209        3,58%         0,3100        2,153        
 160-319           25            282,92        282           283           0,0064        1,09%         0,0100        0,000        
 320-639           0             -             -             -             0,0000        0,00%         -             -            
 640-1279          0             -             -             -             0,0000        0,00%         -             -            
 1280-2559         773           1378,91       1310          1379          0,1967        33,71%        0,4700        2,367        
 2560-5119         411           3123,28       2692          4005          0,1046        17,92%        0,2400        1,104        
 5120 and greater  177           5740,83       5318          9257          0,0450        7,72%         0,1300        0,456        

----------------------------------------------------------------------------------------------------------------------------------
20

I just checked and the network adapter on one of the NAS was set to jumbo frames for some reason so I have set it back. No impact on transfer speed however.