# performance tuning for a 50 Gbps NIC

**URL:** https://forum.syncthing.net/t/performance-tuning-for-a-50-gbps-nic/22565
**Category:** Support
**Created:** [August 7, 2024, 7:10pm UTC](https://forum.syncthing.net/t/performance-tuning-for-a-50-gbps-nic/22565 "2024-08-07T19:10:02Z")
**Posts on this page:** 12
**Page:** 1

<div class="post-metadata">

### Author: ![name-snrl](https://forum.syncthing.net/user_avatar/forum.syncthing.net/name-snrl/32/17080_2.png) [@name-snrl](https://forum.syncthing.net/u/name-snrl)
#### Post date: [August 7, 2024, 7:10pm UTC](https://forum.syncthing.net/t/performance-tuning-for-a-50-gbps-nic/22565/1 "2024-08-07T19:10:02Z")

</div>

# Specs

I have 10 servers with 50 Gbps NIC, 2Tb RAM, and 128 CPU threads and here are iperf3 result:

```plaintext
[SUM] 0.00-10.00 sec 57.6 GBytes 49.5 Gbits/sec receiver

```

# What I tried

Before I started using syncthing, I used to synchronize directories using a nushell script that just ran rsync in parallel, which was close to max speed. I don’t have the exact results, but the time taken to transfer 10 gigabytes of random bytes was roughly as follows:

30sec (rsync) vs. 2m30sec (syncthing).

the file was created using `dd if=/dev/random of=test.dat bs=1G count=10`.

I have tried changing the following global options as described in the [documentation](https://docs.syncthing.net/users/tuning.html#tuning-for-high-performance):

```plaintext
globalAnnounceEnabled = false;
relaysEnabled = false;
setLowPriority = false;
databaseTuning = "large";

maxConcurrentIncomingRequestKiB = 1024 * 1024 * 5;
pullerMaxPendingKiB = 1024 * 512;
blockPullOrder = "inOrder";
copyRangeMethod = "copy_file_range";
numConnections = 10;

progressUpdateIntervalS = -1;
maxFolderConcurrency = -1;
fsWatcherEnabled = true;
copiers = 128;
hashers = 128;
scanProgressIntervalS = -1;
weakHashThresholdPct = 101;
maxConcurrentWrites = 64;
disableFsync = true;

```

setting env vars:

```plaintext
GOMAXPROCS = "128";
GOMEMLIMIT = "100GiB";
GOGC = "200";

```

The above is nix code, I’m using NixOS, but the latest raw config looks like this:

[cfg.xml](https://forum.syncthing.net/uploads/short-url/t1iARiQk3fJtDJCaT4NBn3KY3te.xml) (14.6 KB)

But nothing works, I can’t get past the 1.5Gbps:

 ![gui](https://forum-uploads.syncthingcdn.net/original/3X/1/e/1e688cab9d1632346d6d23ac7d3eb965501833fc.png)

Syncthing was running as systemd service:

```plaintext
[Unit]
After=network.target
Description=Syncthing service

[Service]
Environment="GOGC=200"
Environment="GOMAXPROCS=128"
Environment="GOMEMLIMIT=100GiB"
Environment="LOCALE_ARCHIVE=/nix/store/9dij3pl4dkdxmxhasjw1pa9hzqv4rjlp-glibc-locales-2.39-52/lib/locale/locale-archive"
Environment="PATH=/nix/store/ysqx2xfzygv2rxl7nxnw48276z5ckppn-coreutils-9.5/bin:/nix/store/36rvynxwln7iz0qq3k1v3r1mna8bma8s-findutils-4.9.0/bin:/nix/store/d9xr7s3z0r8rf0ba22q6ilqv68agymdb-gnugrep-3.11/bin:/nix/store/7xwbkzfrs6flyvjyvd23m8r2mlnycinq-gnused-4.9/bin:/nix/store/d9ff8aqv537mlhpinncx6dwc7a5ky6gk-systemd-255.6/bin:/nix/store/ysqx2xfzygv2rxl7nxnw48276z5ckppn-coreutils-9.5/sbin:/nix/store/36rvynxwln7iz0qq3k1v3r1mna8bma8s-findutils-4.9.0/sbin:/nix/store/d9xr7s3z0r8rf0ba22q6ilqv68agymdb-gnugrep-3.11/sbin:/nix/store/7xwbkzfrs6flyvjyvd23m8r2mlnycinq-gnused-4.9/sbin:/nix/store/d9ff8aqv537mlhpinncx6dwc7a5ky6gk-systemd-255.6/sbin"
Environment="STNORESTART=yes"
Environment="STNOUPGRADE=yes"
Environment="TZDIR=/nix/store/jyh52p2cxrjn8r4ywdv2am5pjkj1xcqa-tzdata-2024a/share/zoneinfo"
CapabilityBoundingSet=~CAP_SYS_PTRACE
CapabilityBoundingSet=~CAP_SYS_ADMIN
CapabilityBoundingSet=~CAP_SETGID
CapabilityBoundingSet=~CAP_SETUID
CapabilityBoundingSet=~CAP_SETPCAP
CapabilityBoundingSet=~CAP_SYS_TIME
CapabilityBoundingSet=~CAP_KILL
ExecStart=/nix/store/n45s8j9ngvh1ws8c1m4v404nnw2kjcd0-syncthing-1.27.7/bin/syncthing \
  -no-browser \
  -gui-address=127.0.0.1:8384 \
  -config=/home/user/.config/syncthing \
  -data=/home/user/.config/syncthing \

Group=users
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateMounts=true
PrivateTmp=true
PrivateUsers=true
ProtectControlGroups=true
ProtectHostname=true
ProtectKernelModules=true
ProtectKernelTunables=true
Restart=on-failure
RestartForceExitStatus=3 4
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
SuccessExitStatus=3 4
User=user

[Install]
WantedBy=multi-user.target

```

# The Question

So, my question is: is this even possible? Fully load the bandwidth using syncthing? Relying on this issue seems like it’s not possible at all.

> <https://github.com/syncthing/syncthing/issues/8165>
>
> \## The problem
> Hashing several TB's of data takes a long time and transfers see…m to peak around 1Gbps (Just over 100MBps), even on a 10Gb connection.
> \## Usefulness
> Even though our use case is for huge files between powerful workstations (Xeon 4114 and Xeon 6244 using AES\_128), this could also apply to less powerful devices pushing large files on a good connection.
> \## Workaround
> Our largest files are placed in specific subfolders, which are synced a-\>b and b-\>a each night by a \[custom script\](https://github.com/bovesan/mistika-hyperspeed/blob/master/Tools/Subcp/subcp.py). This relies on regular linux cp command over nfs, as neither rsync or scp seems able to saturate the connection. These files are seldom modified, which makes cp --update (checks file modification time) plenty.
> \## Proposal
> \- Disable hashing for these files and only rely on modification time (should probably make sure time is set correctly across devices).
> \- Allow unencrypted transfers for those files.
> \## References
> https://forum.syncthing.net/t/choosing-cipher-for-data-transfer/9812/3
> https://github.com/syncthing/syncthing/issues/1019

This thread is exactly how I feel right now, but there is no answer.

> [@high performance tuning](https://forum.syncthing.net/t/high-performance-tuning/21920):
>
> Hello! So I’m trying to sync folders between 2 servers in 2 geo locations. here’s some stats for both servers: dual Xeon(R) Gold 6346 CPU @ 3.10GHz 250GB ram 100Gb ethernet card Hashing performance is 879.98 MB/s storage speed on both sides limited to dual 32Gb fibre (getting 6GB/s) relays disabled I can rclone copy the same files between the servers @ 3Gb/s syncthing only getting max 300 MB/s for syncing files. The scanning operation does max out the storage at 6GB/s though, which causes 900% C…

I also read a few other threads but didn’t find anything interesting:

> [@Help Optimizing for 10gbps LAN?](https://forum.syncthing.net/t/help-optimizing-for-10gbps-lan/17101):
>
> Good afternoon! First, thanks to all the Syncthing devs for working on such an awesome project! In the past, I’ve used [Unison](https://www.cis.upenn.edu/~bcpierce/unison/) as my file synchronization solution, but I’m finally switching to Syncthing because it looks better and more mature in just about every respect. My primary use case is a large (but not massive) dataset of about 11 Tb (and growing) full of video data and associated project files/assets: I have two main nodes, each running on FreeBSD with ZFS. These machines are identica…

> [@Hashing performance for bears](https://forum.syncthing.net/t/hashing-performance-for-bears/10013):
>
> I wonder if somebody could explain what the figures for hashing performance mean. I’m a bit baffled. Raspberry Pi model B at mine journalctl --since=04:03:03 --until=04:03:10 --no-pager /usr/bin/syncthing – Logs begin at Mon 2017-05-15 11:36:12 BST, end at Fri 2017-06-23 14:12:51 BST. – Jun 23 04:03:03 pi nice[481]: [3YBZS] INFO: Single thread SHA256 performance is 2.5 MB/s using minio/sha256-simd (2.5 MB/s using crypto/sha256). Jun 23 04:03:05 pi nice[481]: [3YBZS] INFO: Hashing performance w…

> [@Hashing Performance](https://forum.syncthing.net/t/hashing-performance/8840):
>
> I just now noticed this really amazing performance: Is this possible? 1.5 Terabit / Second? source: [https://data.syncthing.net/](https://data.syncthing.net/)

> [@Performance tuning](https://forum.syncthing.net/t/performance-tuning/19208):
>
> I setup to VM in US and Europe on GCP(google cloud) respectively. the speed between the two VM can reach 8Gbps(tested by iperf3). when using syncthing, the highest syncing speed can only get 20MiBps(180Mbps). Are there some way to improve the performance, like increasing threads. thanks, Junhua

---

<div class="post-metadata">

### Author: ![AudriusButkevicius](https://forum.syncthing.net/user_avatar/forum.syncthing.net/audriusbutkevicius/32/277_2.png) [@AudriusButkevicius](https://forum.syncthing.net/u/AudriusButkevicius)
#### Post date: [August 8, 2024, 6:16pm UTC](https://forum.syncthing.net/t/performance-tuning-for-a-50-gbps-nic/22565/2 "2024-08-08T18:16:03Z")

</div>

Did you check what resources you sre constrained by?

I.e., iotop for storage utilisation, top for cpu?

There is quite a lot going on there:

> <https://github.com/syncthing/syncthing/blob/17e60b9e0c340d12b80d015663c058bd54d03557/lib/model/model.go#L1951>

So perhaps running it under a profiler would help. Note that the clients have to do cryptographic verification of the data and might also be part of the bottleneck.

---

<div class="post-metadata">

### Author: ![name-snrl](https://forum.syncthing.net/user_avatar/forum.syncthing.net/name-snrl/32/17080_2.png) [@name-snrl](https://forum.syncthing.net/u/name-snrl)
#### Post date: [August 12, 2024, 12:27pm UTC](https://forum.syncthing.net/t/performance-tuning-for-a-50-gbps-nic/22565/3 "2024-08-12T12:27:53Z")

</div>

> [@AudriusButkevicius](#):
>
> Did you check what resources you sre constrained by?

I don’t think the problem could be system resources. 2x64 cores, 2tb ram should be enough. Plus, the same files are copied much faster with rsync. If resources were the problem, rsync would be about as slow, not 5x faster

p.s. CPU utilization is 30% of one core

---

<div class="post-metadata">

### Author: ![AudriusButkevicius](https://forum.syncthing.net/user_avatar/forum.syncthing.net/audriusbutkevicius/32/277_2.png) [@AudriusButkevicius](https://forum.syncthing.net/u/AudriusButkevicius)
#### Post date: [August 12, 2024, 12:51pm UTC](https://forum.syncthing.net/t/performance-tuning-for-a-50-gbps-nic/22565/4 "2024-08-12T12:51:14Z")

</div>

Rsync and syncthing do different things. One copies files just by copying files, other one uses cryptographic hashes to verify content, maintains and looks up state in a database, etc. You are not comparing apples with apples.

I guess my question was not “do you have a powerful computer”. My question was “did you check iostats, cpu stats, context switch counts, memory paging stats” and all other metrics that might suggest what the bottleneck is.

---

<div class="post-metadata">

### Author: ![name-snrl](https://forum.syncthing.net/user_avatar/forum.syncthing.net/name-snrl/32/17080_2.png) [@name-snrl](https://forum.syncthing.net/u/name-snrl)
#### Post date: [August 12, 2024, 1:10pm UTC](https://forum.syncthing.net/t/performance-tuning-for-a-50-gbps-nic/22565/5 "2024-08-12T13:10:57Z")

</div>

> [@AudriusButkevicius](#):
>
> uses cryptographic hashes to verify content

I generate a single 10GB file, does syncthing split it into parts and hash each one, or only once per file? If only once, shouldn’t the speed be comparable to rsync after the hash is calculated?

> [@AudriusButkevicius](#):
>
> I guess my question was not “do you have a powerful computer”. My question was “did you check iostats, cpu stats, context switch counts, memory paging stats” and all other metrics that might suggest what the bottleneck is.

And yes, I checked iotop(disk usage), htop(cpu usage), bmon(network bandwidth). None of the above showed values close to what they were when using rsync. What else can I check?

---

<div class="post-metadata">

### Author: ![mraneri](https://forum.syncthing.net/letter_avatar_proxy/v4/letter/m/b19c9b/32.png) [@mraneri](https://forum.syncthing.net/u/mraneri)
#### Post date: [August 12, 2024, 1:19pm UTC](https://forum.syncthing.net/t/performance-tuning-for-a-50-gbps-nic/22565/6 "2024-08-12T13:19:06Z")

</div>

> [@name-snrl](#):
>
> I generate a single 10GB file, does syncthing split it into parts and hash each one

Yes. It does.

I think the point is to stop comparing it to rsync. They do different things.

The bottle neck question is relevant to actually UNDERSTANDING why the performance isn’t as fast as you expect so that the experts here can either provide an explanation or possibly suggest some things to relieve that bottleneck and improve the performance.

And in the end if rsync work better for you, just use it.

---

<div class="post-metadata">

### Author: ![acolomb](https://forum.syncthing.net/user_avatar/forum.syncthing.net/acolomb/32/11343_2.png) [@acolomb](https://forum.syncthing.net/u/acolomb)
#### Post date: [August 12, 2024, 1:19pm UTC](https://forum.syncthing.net/t/performance-tuning-for-a-50-gbps-nic/22565/7 "2024-08-12T13:19:22Z")

</div>

Yes, dividing into blocks is a central concept to how Syncthing works: [Understanding Synchronization — Syncthing documentation](https://docs.syncthing.net/users/syncing.html)

---

<div class="post-metadata">

### Author: ![name-snrl](https://forum.syncthing.net/user_avatar/forum.syncthing.net/name-snrl/32/17080_2.png) [@name-snrl](https://forum.syncthing.net/u/name-snrl)
#### Post date: [August 12, 2024, 1:24pm UTC](https://forum.syncthing.net/t/performance-tuning-for-a-50-gbps-nic/22565/8 "2024-08-12T13:24:35Z")

</div>

> [@mraneri](#):
>
> And in the end if rsync work better for you, just use it.

Maybe, but I’m still curious to understand how we can speed up syncthing

---

<div class="post-metadata">

### Author: ![name-snrl](https://forum.syncthing.net/user_avatar/forum.syncthing.net/name-snrl/32/17080_2.png) [@name-snrl](https://forum.syncthing.net/u/name-snrl)
#### Post date: [August 12, 2024, 1:45pm UTC](https://forum.syncthing.net/t/performance-tuning-for-a-50-gbps-nic/22565/9 "2024-08-12T13:45:24Z")

</div>

@AudriusButkevicius @mraneri @acolomb oke, I did the profiling according to [this guide](https://docs.syncthing.net/users/profiling.html).

Idle:

[syncthing-cpu-linux-amd64-v1.27.7-133211.pprof](https://forum.syncthing.net/uploads/short-url/mX4oXoaN7HvmnUTjFtEUvXKNcNb.pprof) (3.1 KB)  
[syncthing-cpu-linux-amd64-v1.27.7-133403.pprof](https://forum.syncthing.net/uploads/short-url/67QxLsjqDusIFLeSFJTILSGIYhT.pprof) (935 Bytes)  
[syncthing-heap-linux-amd64-v1.27.7-133524.pprof](https://forum.syncthing.net/uploads/short-url/bs1qVQOlZclyRd7bVGKL8bou2PL.pprof) (142.9 KB)  
[syncthing-heap-linux-amd64-v1.27.7-133529.pprof](https://forum.syncthing.net/uploads/short-url/n1ikkZTHNTcHKYTvVI7E4OpaeN0.pprof) (143.6 KB)

During sync:

[syncthing-cpu-linux-amd64-v1.27.7-133645.pprof](https://forum.syncthing.net/uploads/short-url/1HBpP0pVFRi6Q6Do6NjgGYlhsNa.pprof) (11.0 KB)  
[syncthing-cpu-linux-amd64-v1.27.7-133723.pprof](https://forum.syncthing.net/uploads/short-url/ySrMgivLRGYDac3CEWFF3lPHJLh.pprof) (10.4 KB)  
[syncthing-heap-linux-amd64-v1.27.7-133758.pprof](https://forum.syncthing.net/uploads/short-url/pIFDrTPNtbCg05GrrhLiprcKjR3.pprof) (142.9 KB)  
[syncthing-heap-linux-amd64-v1.27.7-133803.pprof](https://forum.syncthing.net/uploads/short-url/hjqzPTFEqBIG0DcTU5BKOSK70Za.pprof) (143.1 KB)

---

<div class="post-metadata">

### Author: ![imsodin](https://forum.syncthing.net/user_avatar/forum.syncthing.net/imsodin/32/3317_2.png) [@imsodin](https://forum.syncthing.net/u/imsodin)
#### Post date: [August 12, 2024, 8:20pm UTC](https://forum.syncthing.net/t/performance-tuning-for-a-50-gbps-nic/22565/10 "2024-08-12T20:20:32Z")

</div>

Looks like all those profiles are from the sending side. And they just confirm that the expected things are going on on the cpu, but not much overall. I doubt profiles from the receiving side will add much.  
Also the profile involves `encryptedModel` - generally quite helpful to mention when you use non-standard settings. Doesn’t seem to be the issue here though, as that should mostly impact cpu.

I don’t think you mentioned what the disk setup is. Especially where the database is located.

---

<div class="post-metadata">

### Author: ![name-snrl](https://forum.syncthing.net/user_avatar/forum.syncthing.net/name-snrl/32/17080_2.png) [@name-snrl](https://forum.syncthing.net/u/name-snrl)
#### Post date: [August 12, 2024, 9:19pm UTC](https://forum.syncthing.net/t/performance-tuning-for-a-50-gbps-nic/22565/11 "2024-08-12T21:19:21Z")

</div>

> [@imsodin](#):
>
> I don’t think you mentioned what the disk setup is. Especially where the database is located.

Both sides have Mellanox MT42822 BlueField-2 (NVMe-over-Fabric) controllers with ZFS and encrypted root. I have 10 identical machines and I’m syncing one of them with the others.

---

<div class="post-metadata">

### Author: ![system](https://forum-uploads.syncthingcdn.net/original/2X/0/0b50e0a202b22ae6a67190759e8c868805b8ed9f.png) [@system](https://forum.syncthing.net/u/system)
#### Post date: [September 11, 2024, 9:19pm UTC](https://forum.syncthing.net/t/performance-tuning-for-a-50-gbps-nic/22565/12 "2024-09-11T21:19:55Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.