Slow sync over internet

I’m trying to troubleshoot a slow sync over the internet, but can’t find the bottleneck. Right now, the sync is running at 900 KiB/s. Both endpoints are VMs with 8 cores allocated and 4gb of ram, running windows server 2019. No single core is at more than ~30% usage, and total CPU usage is below 30%. Ram is at about 50% usage. Both the source and destination are freenas arrays, with disk utilization below ~10% for all disks in both arrays. The CPUs in the freenas VMs on both ends are below 25% usage. A speedtest on the first machine is about 150/150 Mbps, and a speedtest on the second is about 50/200 Mbps (right now, this can be higher at times). If I connect to a VPN hosted from the same site as the first machine, an iperf test yields ~50 Mbps. There are two folders syncing from the first to the second, one with lots of large files, and one with a few small files and a few medium-sized ones. I made sure that I’m not connected over any relays as well.

How can I improve performance here?

Checked your connection types? https://docs.syncthing.net/users/faq.html#why-is-the-sync-so-slow

Looks good, both ends say Address <Public IP>:22000

What does iperf show without the VPN? What’s the latency?

I don’t know how to test iperf performance without the VPN and without opening any ports for iperf, but latency is about 30ms.

As a next step I’d try to setup a isolated test case.

Start syncthing with STTRACE=model,connections env var set.

Have a new empty single folder between the devices, add a single large (1GB I guess) file on one side, capture logs from both sides, and post them here.