Meshed syncthing devices synchronize very slowly

User Stories
1

Environment

  • 3 syncthing devices - 2 linux machines, 1 android device
  • all devices have syncthing version 1.9.0
  • I configured one folder, that is shared between all the three devices. This means each device shares this folder to each other connected device. This is what I call “meshed”.
  • Content of the folder: I used the current git master branch content of the linux kernel to test that (~70.000 files with about 912MiB). But I think this content is not the issue, because I had this issue before with different other folders and I just wanted to make a fresh test szenario to reproduce this.

Steps to reproduce for my issue

  1. On [Device 1]

  2. On [Device 2]

    • Allow the shared folder from [Device 1]
    • Look at the download rate: At my desk the complete folder was finished after ~15 minutes (not too fast, but one device has slow Wifi)
    • Wait until it finished

  3. On [Device 1] and on [Device 2]

    • Share the folder to [Device 3]

  4. On [Device 3]

    • Allow the shared folder from [Device 1] and from [Device 2]
    • ISSUE: Look at the download rate: This is now very slow, after half an hour I had synched only about 11% of the content

The sync inside this mesh costallation is always very slow, also after the initial full sync. The issue is not visible any more as soon as you disable all sync paths to one of the three devices. The expected behavior is that the sync in this mesh is as fast as in a two device sync line.

In my opinion the issue could be that there is often a rescan in that “mesh” constellation. Maybe after getting content from one of the devices there is always a rescan to the second device or something like that. But that is only theory, I don’t have a deep understanding of the syncthing internals.

Can you tell me if you officially support that “mesh” configuration? Can you help me to fix it?

If you have further questions, don’t hesitate to contact me.

Thank you!

2

So I don’t think mesh has much todo with this as the downloading behaviour between 1 and N devices is exactly the same. Which is for “me to download part A of file F, what is the device with the least outstanding requests that I should contact”

I suspect whats happening is that one of the devices is just much slower at serving data than the other.

You are saying that if you unshare the folder with one of the devices it all goes back to normal, but have you verified that it goes back to normal if you use either of the two devices?

The dataset in question is not very helpful either, because there is a non-trivial cost in syncthing per file, as every file has to be recorded in the database, fsync’ed (which usually fsyncs the whole disk not just the file), and the kernel source is a ton of tiny files, so 90% of the time is spent fsyncing, oppose to transfering data.

Syncthing by default syncs no more than two files at a time, in your case, the files are so small that in reality the whole file is a single “part” from the above description, so what happens is that a single slow device suddenly slows down the whole pipeline.

Again, this is my suspicion.

I suggest you try with bigger (in the megabytes) files, you’ll that’s much faster and is does not take long even if you share it with all devices (even if one of them is slow) or try removing one device and then the other, to see which one is the cause of slowness.