High file descriptor usage (OpenBSD).

Hi,

I’m not sure if this is new behavior, but my Syncthing nodes running on OpenBSD were maxing out the file descriptor ulimits. I noticed because my CPU would go nuts (bug is already fixed in master it seems).

In the OpenBSD Syncthing package’s README, we used to recommend 4096 fds, but that seems to not be enough for my nodes. I tried doubling this number to 8192, but there is a hard-coded limit of (oddly) 7030 per-process in the OpenBSD kernel. My nodes run OK with this limit, but it seems we are getting close to the max.

I wonder if there’s a way to have Syncthing use fewer fds on OpenBSD? I guess this will involve kqueue details…

Thanks!

Check what the fds are

No clue, sorry. All we do is ask https://github.com/rjeczalik/notify for a recursive watch on folder roots and all I know about that is, that kqueue does not support recursive watches, so notify will walk the folder root and create watches for everything in there (I think independetly from what OS that means watches on all directories, but that might be inotify specific). And all this is true for a long time, afaik nothing changed recently.

Simon, so that’s one file descriptor per file, is it?

So instead of guessing from memory I had a quick look: Notify traverses the tree and watches every directory. So unless kqueue does something more under the hood, that means one fd per directory.

I see. I’ll investigate more, because I don’t think I have 7030 dirs in my shares.

Never mind, it does indeed do something more: Kqueue is a “trigger type watch”, which means it needs to watch both files and directories. So it’s one fd per directory and file. Sorry for piecewise info.

Ah. I see.

This is most unfortunate. So you can never have more than 7030 files in the sum of all of your folders in OpenBSD.

That’s an easily-reached upper limit :frowning:

Good news. The limit can be adjusted with the kern.maxfiles sysctl.

Hi again,

Does unchecking “watch for changes” in the advanced tab of a folder (thus relying on periodic scanning, if I understand correctly) reduce the number of file descriptors required at all?

Presumably Syncthing uses a small fixed number of threads to do periodic scans and doesn’t require every file in a folder to be open a tthe same time (like kqueue)?

Thanks

Sure. In normal scanning it has just one file open at a time per scan. And max concurrent scans is unlimited by default and can be limited in advanced settings.

But surely you try to limit this based upon the number of cores in the machine?

Where is the max concurrent scans option, by the way? I don’t see one in advanced:

Sorry for all of the questions. I’m gathering info to include in the OpenBSD package readme.

Thanks :slight_smile:

It’s in the “very advanced config” in actions>advanced>options>max concurrent scans.

That’s the default limit by folder. I recently wrote a benchmark to see if globally limiting to number of cores with massive parallel hashing is faster, but didn’t find any difference. Apparently the go runtime management goroutines (e.g. regarding thread switching) is intelligent enough for this to have negligible influence. This was CPU only, for spinning disks it’s probably still a good idea to limit parallelism.

No worries, thanks for packaging.

Ah, I’m just reading this:

Seems like go has a smart routine scheduler. So even if you spawn as many go routines as there are files to scan, then you won’t end up with an unbounded number of threads.

That seems to agree with you explanation above :slight_smile:

Cheers

1 Like

Yes and no. This is true in general for cpu bound work, although goroutine switching isn’t completely free (just cheap). However many/most file IO calls are not async, or not reliably available in an async form on all supported platforms, and will consume a thread. One million goroutines attempting simultaneous file io will crash the program due to reaching the thread limit (assuming the fd limit wasn’t reached first). Network io is fine though, one million goroutines waiting on sockets is just a select call somewhere.

1 Like

@calmh I see. But do you agree that in normal Syncthing use-cases, using periodic scans without watchers should use fewer fds?

Sure.

1 Like

So even though there’s no measurable performance improvement on the CPU side (I used data in memory in the benchmark), globally limiting parallel hashing would still be a good thing? I am asking because I have a branch where I am half way there.

I can see it being useful when there are more than 50 folders or so, but at that point you anyway need to not run with the default 60s scan interval and some other changes, so I don’t think it really matters for 99% of users?

By the way, I found this post which explains where the magical 7030 fd limit comes from in OpenBSD:

1 Like