Debugging slow performance


#1

My problem seems to be a recurring one, but I found nothing that could help me in the forum.

I have a simple setup: a PC and a laptop (osx both) within the same LAN network. Probably not relevant, but the outside network connection is about 25MBs. Sync speeds are very low (10s of KB), both for a single large file and for many smaller ones. Sorry, being new I can only upload one picture, but it does not seem that CPU of memory are being stretched on either end.

How can I increase sync speeds?

Thanks


(Audrius Butkevicius) #2

Can you post screenshot from the other side? Also, you’ve blanked out the address, which doesn’t make it obvious if they are connected directly.


#3

Thanks for the quick reply. Uploaded the other side.I blanked out the addresses because I wasn’t sure how safe it was to display them. In any case on both ends they are showing direct contact (i.e. not relaying).


(Audrius Butkevicius) #4

Did you check disk utilization on both sides?


#5

The disks on both sides are SSDs. I have a dashboard with various statistics inc disks - there was nothing extraordinary there.


(Audrius Butkevicius) #6

Can you run both sides with model debugging enabled (as it’s downloading), and provide logs from both sides?


#7

I am running with a ~28MB file on the ‘upload’ side. Here are two log files - the sync is not finished but I guess the picture will be the same later on

ul_side.txt (31.6 KB) dl_side.txt (29.3 KB)


(Audrius Butkevicius) #8

Strange, I only see part of the messages, there are no request response messages. Can you set STTRACE=model environment variable, restart it from the console, and capture the whole log from the console?


(Audrius Butkevicius) #9

Also, I saw this in your logs:

2018-09-09 23:02:02 Device SIJAVPD-F74QNTA-IMBZJ5E-QEOCK4R-DPFHMND-DV3JD4G-A34XZXR-CUFRKQO send rate limit is 2000 KiB/s, receive rate limit is 2000 KiB/s

on both sides.


#10

Regarding the bandwidth limits: I set them up because even while sync never went past 100KB, the whole network was getting hogged. I though this may help. In any case I’d love to get the 2MB limit (for starters at least)!

Other params I’ve changed (based on these forums) but which made no difference: pullerPendingKiB set to 163840, copiers set to 20, and use large blocks set to true.

Here are the logs you requested: ul_side2.log (71.2 KB) dl_side2.log (110.9 KB)


(Audrius Butkevicius) #11

Can you try removing the bandwidth limits? I suspect large blocks + bandwidth limits don’t play ball well, and rate limiting ends up with very spiky throttling. It seems it sends the requests within a second, and then they take 10 or so minutes to arrive.


(Jakob Borg) #12

Those are all weird (apart from large blocks), I suggest setting those things back to default for the purpose of troubleshooting easier. Also,

This also makes no sense and merits further investigation on its own.


#13

Thanks again for all the help!

I’ve removed the copier and puller params. Kept the large blocks. Reapplied the sync test and uploaded the log files again.

ul_side2.log (109.5 KB) dl_side2.log (447.0 KB)

As for the application hogging bandwidth - would like to investigate further. Not sure how


(Clemens) #14

Have you restarted both clients once and seen if the transfer speed increased? I have a problem with sudden slow downs and can fix it by restarting. But this is only a bad work around and does not really fix the problem.


#15

Thanks Clemens. I’ve had several restarts along the way. They did not change anything (which is actually lucky - at least the situation is stable)


#16

I’ve just notices that my previous uploads were w/o removing the bandwidth limits. Here are the updated log files.

BTW, I’ve noticed that when sync starts - bandwidth may go up to 200-300 KBs, but it soon drops to ~50.

ul_side2 2.log (296.4 KB) dl_side2.log (805.1 KB)


#17

Hello:

FWIW, I had a very similar issue last week. Unfortunately I didn’t have time to investigate before I had to implement an alternative sync to meet production deadlines - but I need to revisit it in a couple of weeks.

In my case:

  • This was a two machine cluster, connected over the LAN;
  • Both sides were running macOS 10.12.6;
  • Both sides were running Syncthing 0.14.49;
  • Both sides saw each other on LAN addresses;
  • Both machines had plenty of RAM, CPU and database disk speed to throw at the problem - but it looked like none of those were choking;
  • The disks that the shared folders resided on were running a funny RAID setup, which might have hampered performance - though I would be surprised if that was the root of the problem, considering the slowness I was seeing;
  • There were multiple shared folders configured - about 12 on each side;
  • Both sides had large blocks enabled on all shared folders;
  • Transfers were running ridiculously slowly - it was struggling to sync 20GB over five days in one folder (having paused the other folders to allow it to complete).

Sorry I can’t provide more detail, but I see there’s some common ground with the OP here (macOS, 0.14.49, large blocks, LAN only). I hope to set up a test bench soon, but I won’t have time over the next couple of weeks.

Hope that helps, Pants.


(Audrius Butkevicius) #18

@idobz could you run this build with the same debug options enabled, and provide the logs once more? This adds debug logging around to see where the time is being spent:

https://build.syncthing.net/viewLog.html?buildId=26491&buildTypeId=Syncthing_BuildMac


(Audrius Butkevicius) #19

Actually, just expand the checks from this PR, and download the one that is relevant to you:


#20

I’m sorry, but am kind of illiterate when it comes to github and sons. Can you provide me with explicit instructions?