I am running a syncthing instance on a Raspi 5 which is part of a gluster cluster. I use the glusterfs as storage for syncthing. Without the syncthing
service running, I am perfectly happy with the performance of glusterfs (100+MB/sec), but when I start the syncthing service, the performance drops to almost stall (1MB/sec and less) after some time, which eventually means that syncthing never finishes scanning.
I see almost no network, disk or CPU activity (5+GB free RAM).
A few seconds after I stop the syncthing service, the performance is back to 100+MB/sec.
I have no idea where to start my search for performance, any ideas are welcome.
I’ve posted a similar question to the Gluster mailing list to see if glusterfs might be to blame.
The other two syncthing instances are a win11 in the local LAN and a ceph based linux instance via VPN, the Win11 and the linux-ceph instances sync perfectly well.
What does Syncthing claim it’s doing? (Screenshots are worth a thousand words. Expand the folder in question…) Generally, a scan is made on startup (metadata check), which can take a while for network attached storage. I wouldn’t be surprised if your I/O performance is reduced for the duration; in fact, I’d very surprised if it weren’t. That shouldn’t take 6-8 weeks though.
If the filesystem is case sensitive, checking that box will increase your scan performance significantly.
It claims to scan, those are all static data (it does not make sense to change anything before it reaches a steady state). But with 1MB/sec filesystem performance, the rescan interval of one folder overtakes the scan process of another folder. This screenshot shows three scanning folders, if we wait a few days, we can see four, five or six …
Even if I change nothing, it never reaches a “all up to date” state.
GlusterFS is case-sensitive, the case-sensitive flag ist ticked on all folders.
The syncthing version is the default version that comes with Ubuntu (almost the same one I use with the CephFS). I’ve also thought about switching to a newer/official syncthing version. Maybe I’ll give that a try, but I don’t really like the idea because it makes updating the OS less convenient.
As calmh already pointed out: You are scanning the folder every 4 hours, which according to your description takes a very long time without any changes. That means it’s not actually accessing data, so your typical/achievable throughput on glusterfs doesn’t matter. And it tracks with low resource (cpu, memory) usage, because most likely it’s just doing cheap syscalls that are very slow, as glusterfs is slow on metadata operations. Which isn’t exactly unexpected for any network based filesystem, even more so one designed as “cloud storage”. I was hoping I could link to docs that do recommend to use syncthing on the same device as the storage, not on a networked filesystem, but unfortunately there doesn’t seem to be any - could make a good FAQ, this comes up every now and then.
For completeness, though I doubt it’s related to the issues here: That’s definitely better than glusterfs (or any network based FS), but might still be slow-ish depending on what kind of ZFS setup and what is the backing storage. Generally the faster the storage for the database the better.
The more interesting question, IMO, is if the initial scan can complete when running a supported version.
I don’t have an RPi 5 to do any testing with; I recommend that you start by attempting to sync a much smaller (10% of what you have now?) number of files and see if the initial scan can complete. If it completes within a few days that would be a good start for you to understand what you can expect in terms of performance with this environment.
Ok, I will use strace to find out something about these metadata operations. There are also some performance tuning tipps on the glusterfs website to improve metadata operations.
