TrueNAS / FreeNAS Folder Watcher Unusably Slow, Crashes Host

Good evening! I’m continuing my journey of setting up Syncthing on some medium-sized folders (totaling some 600k files). I initially posted some discussion about optimization for my use case in another thread ( Help Optimizing for 10gbps LAN? ) however I think I’ve identified a different issue, so I’m starting a new thread.

For context, I’m running Syncthing on two different TrueNAS boxes, with some of the smaller folders also being synchronized with two Linux machines and a macOS laptop. On TrueNAS, Syncthing is running in its own BSD Jail, which I manually created and followed the installation instructions to get running. (I did this instead of using the TrueNAS-supplied plugin because I wanted to make sure I was using the latest release of Syncthing.)

One the number of files managed by Syncthing started to exceed 250k, I started noticing performance issues.

Specifically, I noticed that the overall system (outside the jail!) was beginning to slow down dramatically. System processes (like metrics collection) begin to time out, logging in (even at the console) is slow, running basic commands like top and such take forever. Even already-running services like SMB and NFS will time out and clients will get disconnected.

Eventually the system locks up completely (to the point where it will not even shut down cleanly) and it must be powered off.

Troubleshooting was really confusing, because no processes seemed to be consuming much of any system resources, and the performance slowdown would persist even if Syncthing was not visibly active. (Not scanning, not syncing.) However, if I paused all of the folders on the system, the performance issues would clear up.

After trying a few different things I stumbled on the solution: Watching for changes in the folder must be disabled to prevent the system from locking up.

This seems to have been written about in a few other places:

It seems very unclear to me where the root cause of this issue is, whether it’s being caused by a bug within FreeBSD, some issue related to Jails, or whether Syncthing is doing something “wrong” to cripple the system. (Or something completely unrelated.)

However, given the severity of the issue, I wanted to bring it back to the attention of the relevant parties. (I also wanted to provide a little more searchable material that search engines can pick up for the benefit of other hapless souls affected by this issue.)

I also posted at the TrueNAS community forum, hoping to get some attention from those folks: Syncthing Jail with Folder Watcher Unusably Slow, Crashes Host | TrueNAS Community

I can “work around” the issue by simply disabling folder watching, but that’s not super great because I have users submitting changes to these folders via SMB and NFS and I would like these changes to get picked up and synchronized as quickly as possible.

1 Like

Thanks for bringing this up. It’s indeed a known limitation of the BSD file watching mechanism (kqueue). There’s no solution other than disabling filesystem watcher. We even once discussed doing that automatically on BSDs. Either on an arbitrary limit of items in a folder or just generally. I should probably get around to doing that, as it’s not nice to knowingly get users into a state that’s known to be problematic.

How many items (dirs and files) do you have in the affected folders?

Filed an issue to track it and as a reminder: Watching for changes using too many system resources on BSDs (kqueue) · Issue #7855 · syncthing/syncthing · GitHub

1 Like

Thanks for replying @imsodin!

I’ll subscribe and add a couple of thoughts there.

The system has two separate Syncthing instances because of UID/GID constraints. The hierarchy looks like this:

  • Syncthing A
    • Folder A files 248,330 dirs 10,639 ~11.2 TiB
  • Syncthing B
    • Folder B files 339,636 dirs 37,701 ~58.2 GiB
    • Folder C files 5,531 dirs 440 ~40.8 GiB
    • Folder D files 5,443 dirs 440 ~45.5 GiB

The issue really didn’t become apparent until I set up Syncthing B.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.