Hello! I have syncthing transferring files from a folder on my server to a mirror at home. Server is FreeBSD 12 (syncthing v 1.0.0) and mirror is Ubuntu 18.04 (syncthing v1.1.1 64bit). I’m seeing excessive RAM usage on FreeBSD.
The data being synced is 125 GB, 1,319 files and 136 folders. Syncthing was running for a day or so on the FreeBSD machine, using about 85M of RAM for the main process (another process uses 32M and that one doesn’t seem to change). It didn’t have any changes to transfer during that time.
I created a new file of about 2GB in size and I saw syncthing using CPU to scan it (or hash it, or whatever it was doing). Then when syncthing started actually transferring the file, the memory usage increased to 216M! On the linux mirror, the web interface shows about 96M of ram usage.
As I type this a few minutes later it still is using 216M. If I restart syncthing on the server, memory use starts at 64M and as I sit here typing this it has slowly increased to 67M – I’m going to predict it will climb to about 80-90M or so then stop. UNLESS it needs to move files again … then I’m not sure how high it will possibly go.
Is this normal? If so, is the option I saw to change garbage collection only a compile-time options? It didn’t seem to work as a run-time option.
Any other suggestions? I’m running the package that FreeBSD 12 has in their repos. Thanks!
The pull process will try to sync 32MB at once, meaning at least that amount of memory will be used during pull (for every device that is pulling from you). You are seeing an increase of ~130MB compared to idle. That seems like a bit much to me but nothing to worry about. If you want to know what is really going on you can capture a heap profile: https://docs.syncthing.net/users/profiling.html
Keep in mind that 32 MiB of data gets copied several times for encryption, protobuf encoding and sending. Plus GC overhead. So yeah, this is as designed.
After the above, without having restarted Syncthing, there were three files synced in the other direction (mirror to server), average size about 400 Megabytes each. Memory used (server) is still low, 82M and mirror 99M.
So that transfer didn’t trigger much memory usage on the server. Is memory usage higher on the sending machine? Does it use more memory for larger files (the difference between 500MB and 2GB), or as transfers take longer?
Over 200 megabytes of RAM usage is pretty significant… would nearly rule out running on a standard storage VM (VPS) with 512 MB of RAM.
I guess I’ll have to keep watching it, unless there’s a treatise in the docs somewhere I haven’t found yet about what takes memory.
Is garbage collection supposed to eventually bring this >200MB of RAM use back to the idle state of around 85MB? Can the GC options only be changed when compiling?
Thanks for all your hard work, Syncthing is useful for me in a number of ways!
There’s no specific documentation; you can assume that “doing stuff” requires memory, whether that’s scanning, syncing, or serving files to other devices, and that the memory required is at least vaguely related to how much “stuff” needs “doing”. That is, scanning a 100 GB file will require more memory than a 100 MB file, because the block list is larger; syncing one million changed files requires more memory than syncing one changed file because the queue is larger, and so on. Indeed once idle it will be released, at some point.
There is really only one knob to tweak on the Go GC, which is set at runtime and not compilation: the GOGC environment variable. This is, very roughly, an overhead value - larger values means GC will accept higher overhead at the gain of less CPU usage; lower values mean GC will try harder to keep memory usage down, at the cost of more CPU usage. The default is 100, you might want to set it to 50, 25, etc to keep the peak usage down a bit (maybe). (syncthing --help says this too.)
Yes, I did see that but failed to make it work. My updated thought to maybe make it work is to put it in /etc/rc.conf (freebsd) like other variables I have for syncthing:
Hi again… sorry to take your time here. The idea of setting environment variables to affect how a program runs is new to me. I think I found it, my shell is csh, so I added this:
setenv EDITOR vi
setenv PAGER less
setenv BLOCKSIZE K
setenv GOGC=50 syncthing
…which looks promising. But, is root’s .cshrc file the correct one to edit? The init system is starting syncthing at boot, so it seems like root would be the one. But, syncthing runs as www user. But since www user isn’t a real system user account, I don’t think there would be any .cshrc file for www.
Am I on the right track? Is there a way to ask syncthing, while running, to dump the options it is currently using?
Thanks! It’s kind of fun to hit the edges of my knowledge, though I know it makes me look a fool!
Unfortunately, I think we (I, at least) are not the people to properly guide you here. My example above works to do the thing in a terminal using bash or sh; perhaps start there to experiment.
A quick test yields a positive outcome: I created a new 2GB test file on the server. After hashing for a couple minutes with pretty high (but perfectly acceptable) CPU use, it began to transfer the file. Ram use (as observed in TOP, the RES column) increased from about 58M to about 148M, which is a good improvement compared to over 200M.
I’m happy! Thanks again.
ps, I wish I could edit posts on this forum. It tells me the body is too similar. I had misytped GOGC as GOCG in my earlier post.
also, I deleted a post, and reposted, but I don’t see the repost. So here’s what I did that seems to have worked.
Edit the rc file
/usr/local/etc/rc.d/syncthing
add this line
export GOGC=50
restart syncthing. Note, package upgrades will probably overwrite this change! There would need to be a hook to add environment variables in rc.conf but I don’t think that is currently supported.
Thanks for the correction.