On lan: sudden speed drops from 100 MiB/s to 10 MiB/s

Hi everybody,

thanks for this great software! I really like the ease of use combined with all the possibilities especially the file encryption (untrusted devices).

I have the following setup at home:

  • Hardkernel Odroid N2
  • the Odroid N2 has the newest Armbian 23.02.2 Jammy (Linux 6.1.11-meson64) installed
  • the Armbian is installed on a Samsung 860 Evo SSD connected via USB 3.0 of the N2. The syncthing target folder of the N2 is also on this SSD.
  • for this test my Notebook (Fedora 38, Ryzen 6600U) and the N2 are both connected with a router by cable
  • Notebook and N2 both have Syncthing 1.23.4 installed
  • On Syncthing startup on the N2 it displays

Single thread SHA256 performance is 978 MB/s using minio/sha256-simd (885 MB/s using crypto/sha256)

Hashing performance is 354.89 MB/s

  • On the N2 I set “Set low priority” to false (not checked in the GUI). Syncthing_Set_low_priority

With iperf3 I get the 935 Mbit/s between Notebook and N2, so the cable connection seems to be fine.

On the Notebook I have a picture folder with 146 GiB. This folder is now shared with the N2 and the N2 is defined as an untrusted device. I entered a password and set the N2 as untrusted device.

When the syncing starts everything is super fast, I see a speed between 90 MiB/s and 110 MiB/s. It keeps this speed for several minutes. After some minutes the speed suddenly drops to around 10 MiB/s.

Then I paused the syncing and resumed it, but it stays at 10 MiB/s.

When I restart syncthing on the N2 it sometimes starts again with 100 MiB/s and after some time it drops again to the 10 MiB/s.

With sudo htop on the N2 I see that Syncthing is using all 6 cores 15-80 % when the speed is 100 MiB/s.

After the speed drop Syncthing seems to use the CPU a lot less. At both times the CPU temperature of the N2 is fine with around 42° C.

Does anybody has an idea what the problem could be? Are there any tweaks or changes to the settings which could prevent the speed drop?

Thanks for your help!

(Output from the lscpu command in Armbian on the ODROID-N2 would be very helpful.)

Based on the details provided so far along with reviewing the product page and datasheets, my initial thoughts are…

big.LITTLE

The CPU in the ODROID-N2 was intended for media streaming devices (e.g., Amazon Fire TV Cube) rather than as a general purpose server CPU.

With the ODROID-N2 configured as an untrusted device in Syncthing, it’s got quite a bit more work to do compared to the notebook with the Ryzen 6600U APU.

The ODROID-N2 is equipped with an Amlogic S922X ARM-based CPU. Although there are 6-cores, they aren’t all equal. It’s a big.LITTLE architecture that pairs a quad-core ARM Cortex-A73 (debuted in 2016) with a more energy efficient – and much less powerful – dual-core ARM Cortex-A53 (debuted in 2012).

Comparing the Amlogic S922X (top) to the quad-core Broadcom BCM2711 (bottom):

(It’s not exactly an apples-to-apples comparison because the results are from Android on the S922X vs. Linux on the BCM2711, but Android does use a Linux kernel.)

So a hexa-core ODROID-N2 is just a little over 1/2 the speed of a quad-core Raspberry Pi 4 in single-core and multi-core benchmarks. Although the ARM Cortex-A73 in the ODROID-N2 is a tad bit faster than the ARM Cortex-A72 in the RPi 4, the RPi 4 was designed for general use in desktops, servers, etc.

As Syncthing forks processes and spawns threads, some of those processes and threads are going to end up on the dual-core Cortex-A53, causing dramatic ups and downs in overall performance. Same issue for the OS. Ambian needs to be tuned to account for the mixed CPU architecture. Tweaking the Linux scheduler settings, using the taskset command, and/or some other method to optimize use of the various CPU cores is required for maximum system performance.

### Untrusted device

According to the Amlogic S922X datasheet, only the Cortex-A73 portion implements the ARMv8-A architecture required for AES-NI support (note that the cryptographic extensions are optional). But even if AES-NI is included, I don’t know if Go currently supports it on ARM.

If encrypting the files from your pictures folder is absolutely required, then consider encrypting them before syncing to the ODROID-N2 because your notebook has significantly more horsepower. A per-file based encryption such as gocryptfs, EncFS or securefs would work well.

RAM

The ODROID-N2 is equipped with 4GB of DDR4 RAM. If there’s no desktop environment and Syncthing is the only service besides OpenSSH, then it should be fine. Otherwise some system tuning will be required to avoid excessive swapping to virtual memory.

How many files are contained in your 146 GiB pictures folder will also impact overall sync performance (Syncthing’s database will be bigger, inotify needs memory for its event queue, etc.).

Syncthing

Refer to Syncthing’s tuning documentation page for suggestions on optimizing Syncthing for a particular environment.

There’s been a similar report in the past about Syncthing that we didn’t resolve, so I’m not ruling out a bug of some sort. Pinpointing the slowdown point in the logs, running with at least -verbose might perhaps give some hints.

The untrusted feature does encryption on the trusted host (otherwise there’d be not much point in encryption) so the untrusted device has actually less to do/hash than a normal one.

D’oh! Thanks @imsodin for pointing out my blunder. :crazy_face:

The Odroid N2 is in general more powerful than the RPi 4B (which has no aes hardware support).

But as imsodin already pointed out it seems that all the encryption power isn’t even needed for an “untrusted device”.

I removed the picture folder (which has 33784 files and 488 folders) and added it again, so the whole transfer started again.

This time it transferred 130 GiB at full speed and then the speed drop to 10 MiB/s showed up. Had it running with -verbose, but I couldn’t identify anything. Are the logs saved anywhere else so that I can look at them again?

Nevertheless after some minutes I just resumed the transfer and the transfer speed was back to full speed until everything was transfered.

Could this be sth. related to the SSD? But based on tomshardware review even if the SSD Cache is used up the 860 Evo shouldn’t become slower than 280 MB/s.

This sounds like thermal throttling to be honest.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.