syncthing using up, all system resources ubuntu/ filling ressource limit

Niclas · January 1, 2025, 8:57pm

Hi, I am running syncthing on a server I rent to sync a few folders, between my devices. I am running syncthing in a screen, on the server. The server is running

OS: Ubuntu 22.04.5 LTS x86_64 Kernel: 5.2.0-1160.105.1.vz7.214.3

and I use the server for some other things too. One thing, that in the past was severly disturbed by syncthing using too many system ressources, is a minecraft server. But I can’t start it. Because

[Server thread/ERROR]: Encountered an unexpected exception java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

But as soon as I restart syncthing through the web ui, the server starts without issues.

In the past, I as far as I remembered I had an error somewhere else, since syncthing had the maximum ammount of files open.

So somehow it seems to accumalate over runtime with syncthing. Running Syncthing v1.28.1, Linux (64-bit Intel/AMD Container)

Is there a setting I can tweak or a starting argument I can give?

I start syncthing with the command

screen -d -m syncthing serve --home="<userpath>/syncthing" --gui-address="http://0.0.0.0:8384"

(replaced my exact user path to )

calmh · January 1, 2025, 9:59pm

What’s the output of running ulimit -a in shell on that server?

Niclas · January 1, 2025, 11:17pm

user@server ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) 0
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 4124744
max locked memory           (kbytes, -l) 1048576
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 62987
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

calmh · January 2, 2025, 8:14am

Looks reasonable. I suspected there could be a low limit on the number of threads as this is some sort of openvz setup? That does not look like it’s the case so I don’t know.

bt90 · January 2, 2025, 9:27pm

Did you rule out that you’re not actually running out of memory on your host?

Niclas · January 3, 2025, 10:38am

Yes, I am renting the server from local Company.

this is the output of top, in a state where some Ressource has run out

top - 11:29:06 up 29 days, 22:57,  4 users,  load average: 0,07, 0,07, 0,06
Tasks:  60 total,   1 running,  59 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0,3 us,  0,2 sy,  0,0 ni, 99,4 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
MiB Mem :   8192,0 total,   7073,0 free,    575,4 used,    543,6 buff/cache
MiB Swap:      0,0 total,      0,0 free,      0,0 used.   7616,4 avail Mem

I just now tried to start the mincraft server to see what would happen/if the problem is there. I had multible “./start.sh: fork: retry: Resource temporarily unavailable” and then after exiting the screen and trying again

[0.003s][warning][os,thread] Failed to start thread "G1 Main Marker" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 4k, detached.
Error occurred during initialization of VM
Could not create ConcurrentMarkThread
[0.004s][warning][os,thread] Failed to start thread "G1 Conc#0" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 4k, detached.
[0.005s][error  ][gc,task  ] Failed to create worker thread

calmh · January 3, 2025, 11:14am

All of that is consistent with the number of threads/processes being artificially limited to a low number, it’s just not visible in your ulimit output

bt90 · January 3, 2025, 11:21am

A shot in the dark, but maybe it’s caused by a systemd unit. LimitNPROC is a huge footgun once containers are involved…

github.com/caddyserver/dist

Switch systemd unit to TasksMax

opened 03:20PM - 28 Dec 23 UTC

bt90

I'm hitting the same problem as outlined in https://github.com/caddyserver/caddy…/issues/1802. The culprit seems to be how systemd handles the `LimitNProc` option: https://github.com/caddyserver/dist/blob/49a805b0196e8c9e394cfe3546f2cd568d6e37d1/init/caddy.service#L30 While caddy doesn't occupy that many processes, some other docker containers seem to use the same UID for their processes: ```sh sudo ps -U caddy ``` ``` PID TTY TIME CMD 4491 ? 00:00:01 mailrise 36706 ? 00:00:28 postgres 36760 ? 00:00:01 postgres 36761 ? 00:00:06 postgres 36762 ? 00:00:10 postgres 36763 ? 00:03:55 postgres 36764 ? 00:00:14 postgres 36765 ? 00:01:17 postgres 36766 ? 00:00:00 postgres 1597030 ? 00:00:03 postgres 1599669 ? 00:00:03 postgres 2081581 ? 00:25:43 redis-server 2082548 ? 00:00:36 postgres 2082623 ? 00:00:34 postgres 2654461 ? 00:00:00 start.sh 2654495 ? 00:00:00 Xvfb 2654496 ? 00:00:00 dumb-init 2654497 ? 00:48:58 node 2654671 ? 00:01:16 chrome 2654672 ? 00:01:16 chrome 2654673 ? 00:01:14 chrome 2654674 ? 00:01:14 chrome 2654675 ? 00:01:16 chrome 2654676 ? 00:01:13 chrome 2654677 ? 00:01:15 chrome 2654678 ? 00:01:14 chrome 2654683 ? 00:00:00 chrome_crashpad 2654684 ? 00:00:00 chrome_crashpad 2654685 ? 00:00:00 chrome_crashpad 2654686 ? 00:00:00 chrome_crashpad 2654691 ? 00:00:00 chrome_crashpad 2654692 ? 00:00:00 chrome_crashpad 2654693 ? 00:00:00 chrome_crashpad 2654694 ? 00:00:00 chrome_crashpad 2654703 ? 00:00:00 chrome 2654704 ? 00:00:00 chrome 2654705 ? 00:00:00 chrome 2654706 ? 00:00:00 chrome 2654707 ? 00:00:00 chrome 2654708 ? 00:00:00 chrome 2654709 ? 00:00:00 chrome 2654710 ? 00:00:00 chrome 2654711 ? 00:01:14 chrome 2654712 ? 00:01:13 chrome 2654715 ? 00:00:00 chrome_crashpad 2654717 ? 00:00:00 chrome_crashpad 2654718 ? 00:00:00 chrome_crashpad 2654722 ? 00:00:00 chrome_crashpad 2654723 ? 00:00:00 chrome 2654724 ? 00:00:00 chrome 2654727 ? 00:00:00 chrome 2654728 ? 00:00:00 chrome 2654729 ? 00:00:00 nacl_helper 2654730 ? 00:00:00 nacl_helper 2654732 ? 00:00:00 chrome_crashpad 2654750 ? 00:00:00 chrome_crashpad 2654752 ? 00:00:00 chrome_crashpad 2654753 ? 00:00:00 nacl_helper 2654757 ? 00:00:00 nacl_helper 2654759 ? 00:00:00 chrome_crashpad 2654761 ? 00:00:00 chrome 2654762 ? 00:00:00 chrome 2654767 ? 00:00:00 chrome_crashpad 2654768 ? 00:00:00 chrome_crashpad 2654770 ? 00:00:00 nacl_helper 2654781 ? 00:00:00 chrome 2654786 ? 00:00:00 chrome 2654796 ? 00:00:00 chrome_crashpad 2654800 ? 00:00:00 chrome 2654802 ? 00:00:00 chrome 2654816 ? 00:00:00 chrome_crashpad 2654817 ? 00:00:16 chrome 2654818 ? 00:00:17 chrome 2654821 ? 00:00:00 chrome 2654822 ? 00:00:00 chrome 2654823 ? 00:00:17 chrome 2654824 ? 00:00:17 chrome 2654828 ? 00:00:00 nacl_helper 2654881 ? 00:00:17 chrome 2654884 ? 00:00:00 nacl_helper 2654885 ? 00:00:16 chrome 2654886 ? 00:00:17 chrome 2654901 ? 00:00:00 nacl_helper 2654907 ? 00:00:17 chrome 2654910 ? 00:00:17 chrome 2654916 ? 00:00:00 nacl_helper 2654922 ? 00:00:17 chrome 2654985 ? 00:00:19 chrome 2654999 ? 00:00:00 nacl_helper 2655029 ? 00:00:05 chrome 2655048 ? 00:00:17 chrome 2655053 ? 00:00:05 chrome 2655063 ? 00:00:16 chrome 2655065 ? 00:00:17 chrome 2655066 ? 00:00:17 chrome 2655079 ? 00:00:17 chrome 2655080 ? 00:00:16 chrome 2655085 ? 00:00:05 chrome 2655089 ? 00:00:05 chrome 2655092 ? 00:00:17 chrome 2655096 ? 00:00:05 chrome 2655097 ? 00:00:05 chrome 2655105 ? 00:00:05 chrome 2655129 ? 00:00:05 chrome 2655136 ? 00:00:05 chrome 2655179 ? 00:00:05 chrome 2655180 ? 00:00:20 chrome 2655186 ? 00:00:17 chrome 2655199 ? 00:00:05 chrome 2655223 ? 00:00:05 chrome 2655315 ? 00:00:05 chrome 2655323 ? 00:00:05 chrome 2655330 ? 00:00:05 chrome 2655337 ? 00:00:05 chrome 2655341 ? 00:00:05 chrome 2655346 ? 00:00:05 chrome 2655385 ? 00:00:05 chrome 2655391 ? 00:00:05 chrome ``` The systemd documentation notes that `TasksMax` should be preferred over `LimitNProc`: > Note that `LimitNPROC=` will limit the number of processes from one (real) UID and not the number of processes started (forked) by the service. **Therefore the limit is cumulative for all processes running under the same UID.** Please also note that the `LimitNPROC=` will not be enforced if the service is running as root (and not dropping privileges). Due to these limitations, `TasksMax=` (see [systemd.resource-control(5)](https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html#)) is typically a better choice than `LimitNPROC=`. https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html#Process%20Properties

jakasspeech31 · January 4, 2025, 7:05am

Niclas:

Hi, I am running syncthing on a server I rent to sync a few folders, between my devices. I am running syncthing in a screen, on the server. The server is running

OS: Ubuntu 22.04.5 LTS x86_64 Kernel: 5.2.0-1160.105.1.vz7.214.3

and I use the server for some other things too. One thing, that in the past was severly disturbed by syncthing using too many system ressources, is a minecraft server. But I can’t start it. Because

[Server thread/ERROR]: Encountered an unexpected exception java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

But as soon as I restart syncthing through the web ui, the server starts without issues.

In the past, I as far as I remembered I had an error somewhere else, since syncthing had the maximum ammount of files open.

So somehow it seems to accumalate over runtime with syncthing. Running Syncthing v1.28.1, Linux (64-bit Intel/AMD Container)

Is there a setting I can tweak or a starting argument I can give?

I start syncthing with the command
screen -d -m syncthing serve --home="<userpath>/syncthing" --gui-address="http://0.0.0.0:8384"
(replaced my exact user path to )

Syncthing’s a resource hog. Fix it with ulimit and fs.inotify.max_user_instances. Minecraft deserves better

imsodin · January 4, 2025, 1:19pm

It is?!

Niclas · January 10, 2025, 8:32am

I thaught syncthing was relatively light weight, and I could run everything. But I guess that’s not the case.

I think I will look to manage my shares always syncing otherwise, than to have syncting running on the server. I had the Idea of using another user for it, with tight ulimites, but I currently don’t really have the time to learn and set that up.

Thank you for your help

calmh · January 10, 2025, 10:34am

ulimit has no bearing on a program’s resource usage - it just crashes the program if it tries to exceed the limit.