Good evening! I’m continuing my journey of setting up Syncthing on some medium-sized folders (totaling some 600k files). I initially posted some discussion about optimization for my use case in another thread ( Help Optimizing for 10gbps LAN? ) however I think I’ve identified a different issue, so I’m starting a new thread.
For context, I’m running Syncthing on two different TrueNAS boxes, with some of the smaller folders also being synchronized with two Linux machines and a macOS laptop. On TrueNAS, Syncthing is running in its own BSD Jail, which I manually created and followed the installation instructions to get running. (I did this instead of using the TrueNAS-supplied plugin because I wanted to make sure I was using the latest release of Syncthing.)
One the number of files managed by Syncthing started to exceed 250k, I started noticing performance issues.
Specifically, I noticed that the overall system (outside the jail!) was beginning to slow down dramatically. System processes (like metrics collection) begin to time out, logging in (even at the console) is slow, running basic commands like top and such take forever. Even already-running services like SMB and NFS will time out and clients will get disconnected.
Eventually the system locks up completely (to the point where it will not even shut down cleanly) and it must be powered off.
Troubleshooting was really confusing, because no processes seemed to be consuming much of any system resources, and the performance slowdown would persist even if Syncthing was not visibly active. (Not scanning, not syncing.) However, if I paused all of the folders on the system, the performance issues would clear up.
After trying a few different things I stumbled on the solution: Watching for changes in the folder must be disabled to prevent the system from locking up.
This seems to have been written about in a few other places:
- FreeBSD directory watching causing issues | TrueNAS Community
- Directory watcher on FreeBSD Issue
- syncthing causes whole system to be really unresponsive when using "Watch for changes" on FreeBSD (FreeNAS) · Issue #6531 · syncthing/syncthing · GitHub
It seems very unclear to me where the root cause of this issue is, whether it’s being caused by a bug within FreeBSD, some issue related to Jails, or whether Syncthing is doing something “wrong” to cripple the system. (Or something completely unrelated.)
However, given the severity of the issue, I wanted to bring it back to the attention of the relevant parties. (I also wanted to provide a little more searchable material that search engines can pick up for the benefit of other hapless souls affected by this issue.)
I also posted at the TrueNAS community forum, hoping to get some attention from those folks: Syncthing Jail with Folder Watcher Unusably Slow, Crashes Host | TrueNAS Community
I can “work around” the issue by simply disabling folder watching, but that’s not super great because I have users submitting changes to these folders via SMB and NFS and I would like these changes to get picked up and synchronized as quickly as possible.