Syncthing crippling internet speeds?

I’m putting this out there to test the water as this is only seemed to have started recently.

I use St to pull down monthly backups to offsite. Has been like this for several years, however over the last few weeks there’s been a general complaining of slow internet from different offices. I think it’s St.

I know that the upload speed is going to play a part in the overall drop in the office’s connections, however this is effecting the download speed considerably. Example, this office has a 70/20 connection so the speed was

image

When I remoted into their server. St upload to me was 13Mbs

I pause St and now get

image

I am fully aware that the upload speed was going to be less due to the syncing, but why was the SO ends download speed so badly effected? St didn’t show any significant downloading, but I can’t accept that it was coincidence that within seconds of St being paused the internet went back up to full speed.

I will monitor and watch what’s happening once St starts syncing again as it’s on preparing due to it being a 1.2Tb image.

One thing I will add, I have a 500/70 connection on the RO end so I could really pull down the SO ends download speed if there is a problem

Packets need to flow in both directions, a download results in a fair amount of acknowledgment packets in the other direction.

I understand there’s going to be ‘request traffic’, but I was seeing nearly four times the amount of data being requested over what was actually being downloaded.

Is it possible that what I am seeing has any relation to Slow sync rate on LAN as that would have fast connectivity both ways on a LAN? I’m thinking that too many requests flood the network and leave little bandwidth for data.

Not sure how that thread is related to your issue.

I suspect the 70/20 is 70/20 in isolation, as in, even in a pure download scenario your download rate will drop, even if you are just using 10 out of the 20 upload for something else.

I always read those numbers as 70/0, 0/20.

I guess you can set rate limits on syncthing and see if that’s true/it helps.

Given you are pulling down large images, I doubt its any sort of additional per file chatter that you are referring to.

You could try disabling temp indexes, but I doubt it’s that.

There’s definitely something going on. Site reported slow download speeds as the sync has restarted, so was getting

image

The instant I paused the upload I get

image

There’s nothing in any of the network tests that show why there is so much being downloaded, however I wonder if the QUIC protocol is the cause as maybe the resource monitor only see’s TCP.

Is there an option to disable quic and have just a tcp connection?

I should add, I am only concerned with the sites download speed. Their upload is to be expected as they are send only

You can disable QUIC by setting the listening address on both sides to tcp://:22000, then there is no QUIC to talk to. I can see QUIC connections being somewhat aggressive about filling the upstream, and that in turn choking downloads as well. QUIC isn’t made to back off from congestion like TCP is.

They are quite similar in that regard. See RFC9002.

1 Like

You’re right, I misremembered. Looks like it does take up a little more space than a regular TCP connection, but shouldn’t be completely antisocial at least.

Another office started a large image transfer so has given me a chance to see if the speed issue is there, it is.

image

so I enabled throttling at the receiving end to 900 to see if it would slow down the requesting data…

image

but made no difference. So paused the sync and retested the speed…

image

Can anyone think of a tool I can use that will look at realtime transfer speed on the quic protocol?

I tried tcp://:22000 at both ends, same result. Very slow download speeds on a send only site

bit of a long shot, how hard would it be to obtain a custom build where quic is fully disabled for testing purposes?

Alternatively, I could go back to an earlier version where quic wasn’t so heavily used, if I know which version to look for

If you are not connected using QUIC, you’re not using QUIC. I don’t see how removing the code from the binary would change anything?

I’m leaning towards quic being the cause simply because I see a whole load of bandwidth being used but the resource monitor doesn’t show that same amount of data passing though the network card. I am assuming that quic is invisible to the resource monitor.

It’s also possible that quic is fine and there’s something going on in St where the requests are four times the bandwidth than the download. I admit, this is also very unlikely.

This might be as well caused by your ISP. Quite a few tend to throttle filesharing which is often only equated with high UDP traffic.

It’s never been an issue until maybe the last month? I know that various offices had complained of speeds, but at the time I never thought it was St. It’s only in the last week or so that I have tuned into the issue.

So started experimenting on the Send Only end. Setting any limits to the incoming rate makes no difference to the incoming internet speed. However, if I add values to the outgoing rate limit, I get the following internet incoming speeds

900 = 68.9Mbps

1800 = 66.4

1900 = 43.1

1950 = 11.4

So what I am seeing, the outgoing St transfer speed is having an effect on the incoming internet speed. So if I throttle it hard, so say 7.37Mbps limit, the office has full internet. But increase the limit, the incoming internet speed reduces. So by setting 1950, that’s a limit of 16 Mbps which is close to the offices maximum outgoing internet (they are on 70/20), it almost wipes out all the incoming internet.

This is feeling like a St bug as I can now demonstrate St is having a direct impact on incoming traffic.

I will leave the site on 1800 Outgoing rate limit as that feels like a good balance.

Again, you can’t get incoming traffic without a corresponding outgoing stream of requests and ACKs. It is unsurprising that incoming rates are reduced when you have lots of outgoing traffic. I’m not sure what the bug on our side would even theoretically be, there isn’t a special kind of internet-speed crippling traffic we could be using by mistake.

1 Like

I accept there is going to be request traffic, but it’s the volume of data / requests that bothers me.

A send only site should not be receiving almost 4 times the bandwidth requesting data than is actually being sent?

I wouldn’t expect that either but the only syncthing screenshot I can see when scrolling up shows nothing like that?

1 Like

The screen shots show a before and after speed tests. So in both sets, the office download speed was very low, eg., 6.25 of 70Mbps (As I was uploading, the upload speeds are to be expected). But pause a folder, and the download speeds raise to around 69.

The second test was 13.7, and went up to 57.1Mbps

Windows resource monitor / networking tab doesn’t show why there’s was heavy bandwidth which is why I wondered if it was something to do with QUIC, given it’s UDP based. Syncthing doesn’t show at either end the amount of traffic being generated, only what’s actually being transferred.

So the basis of this thread is that St on ultra fast internet and maybe on lan to lan syncing is generating far more request or hidden traffic than would be reasonably expected.