I’m not sure if this is new behavior, but my Syncthing nodes running on OpenBSD were maxing out the file descriptor ulimits. I noticed because my CPU would go nuts (bug is already fixed in master it seems).
In the OpenBSD Syncthing package’s README, we used to recommend 4096 fds, but that seems to not be enough for my nodes. I tried doubling this number to 8192, but there is a hard-coded limit of (oddly) 7030 per-process in the OpenBSD kernel. My nodes run OK with this limit, but it seems we are getting close to the max.
I wonder if there’s a way to have Syncthing use fewer fds on OpenBSD? I guess this will involve kqueue details…
No clue, sorry. All we do is ask https://github.com/rjeczalik/notify for a recursive watch on folder roots and all I know about that is, that kqueue does not support recursive watches, so notify will walk the folder root and create watches for everything in there (I think independetly from what OS that means watches on all directories, but that might be inotify specific). And all this is true for a long time, afaik nothing changed recently.
Never mind, it does indeed do something more: Kqueue is a “trigger type watch”, which means it needs to watch both files and directories. So it’s one fd per directory and file. Sorry for piecewise info.
It’s in the “very advanced config” in actions>advanced>options>max concurrent scans.
That’s the default limit by folder. I recently wrote a benchmark to see if globally limiting to number of cores with massive parallel hashing is faster, but didn’t find any difference. Apparently the go runtime management goroutines (e.g. regarding thread switching) is intelligent enough for this to have negligible influence. This was CPU only, for spinning disks it’s probably still a good idea to limit parallelism.
Yes and no. This is true in general for cpu bound work, although goroutine switching isn’t completely free (just cheap). However many/most file IO calls are not async, or not reliably available in an async form on all supported platforms, and will consume a thread. One million goroutines attempting simultaneous file io will crash the program due to reaching the thread limit (assuming the fd limit wasn’t reached first). Network io is fine though, one million goroutines waiting on sockets is just a select call somewhere.
So even though there’s no measurable performance improvement on the CPU side (I used data in memory in the benchmark), globally limiting parallel hashing would still be a good thing? I am asking because I have a branch where I am half way there.
I can see it being useful when there are more than 50 folders or so, but at that point you anyway need to not run with the default 60s scan interval and some other changes, so I don’t think it really matters for 99% of users?