Hi, I am setting up syncthing on a router with 512M RAM and it is currently in the process of synching ~9K files (50G in size). The “RAM Utilization” is staying at 230M, which is not a big deal yet as “free” still shows 150M available. I have not been able to find any definitive guidance on how much RAM is required and if it depends on the number and size of files. Is there a way to estimate the amount of memory required? If it stays under 256M, I am fine with that. If it grows I would like to understand the limit. Thx in advance.
RAM usage typically balloons during intensive operations and then settles down to <100 MB during idle, more or less regardless of the number of files. In my experience at least, there are other cases out there that I can’t explain. My regular file server with about 1TB of data in 150k files uses about 55 MB when idling, 80 MB when I open the GUI.
One of the most RAM intensive processes is the first scan of a folder. The reason is that to be able to provide the progress indicator it buffers the names and sizes of everything to scan, which may be a lot of data if there are a lot of files. (This can be disabled by setting progressUpdateIntervalS to -1.) Subsequent scans are less resource heavy.
Similarly, the first time a device is connected and all index information needs to be exchanged will probably be a small spike in memory usage.
I’m still running syncthing on a raspberry pi 1 with 512MB RAM, I think I had a crash once due to out of memory (during initial scan + index exchange or something like that), but it just started again and continued after that. In normal operation there are absolutely no problems with it except that it is quite slow due to the slow CPU.
1 Like
The CPU is not a problem here (dual core ARM @ 1.7GHz), but the memory could be. It is still synching, but “RAM Utilization” is @ 95M. This is awesome for my use case.
Question about (%)VSZ. It shows more RAM than there is available. Is that the reserved, but not allocated? If yes, what would make synching try to use it all? That would crash my router…
PID PPID USER STAT VSZ %VSZ %CPU COMMAND 1575 1568 syncthin S 784m 167% 9% /mnt/data/syncthing-linux-arm-v0.14.2
VSZ is the size of the virtual memory space. It’s a convenient technical fiction used in memory management, nothing more.
Thank you. I will not worry about it then.
Just to explain it a bit more. Let’s have a look at a 32 bit linux kernel and how sets up the memory layout of a running program:
Since we have paging enabled on modern operating systems, the CPU never ever sees physical memory addresses. It always sees virtual addresses and the MMU (which is part of the CPU) translates these addresses into its physical representation.
So, the kernel runs in its virtual address space (it actually maps the whole phsyical memory into virtual memory) and each process runs in its own virtual address space. When ASLR is disabled, the virtual address space of a running program always looks the same. There is:
- the text segment: it includes the memory mapped file of the executable file
- the data segment: static variables, like strings
- the bss segment: there are unitialized variabled, like pointers
- the kernel: It avtually maps itself in the address space of a program in order to enable jumping in kernel space for system calls.
- the stack: currently active functions allocate stackframes here
- the heap: memory which gets allocated by
setbrk() - the mmap segment: Where shared libraries and memory mapped files live
The advantage of this concept is, that every segment can live in virtual reality. The kernel can do demand paging. That means, it can allocate a really large region somwhere on the heap or in the mmap segment and the program thinks: hey I got really much memory, awesome. In real reality, the kernel allocates physical memory when its needed.
The go language adds some fancy abstraction over these low level concepts. When we do some hashing of large files or allocate buffers for something, then the memory is virtually allocated but the kernel can decide when to actually allocate physical RAM. So, just ignore the virtual memory column and look at the RSS column to find out how much RAM syncthing uses. The virtual memory column in htop actually is the sum of all blue reagions in the picture above; RSS is the sum of the actually allocated pages.
1 Like
One more Go-specific aside… When Go returns memory to the operating system, it does so by just advising the OS that it doesn’t need the pages any more. This means that the OS will reclaim and reuse those pages when it needs to - even for things like buffer cache and so on. But until that happens those pages will still remain in memory, and be counted as RSS. So RSS does not necessarily mean “in use” for Go - some of those pages can be pulled back by the OS whenever it wants to.
The “RAM usage” counter in the Syncthing GUI takes this into account, because it knows how many pages have been marked MADV_DONTNEED. top on the other hand doesn’t know this.
2 Likes
Measuring memory usage is far more complex than you think. More news at 10!
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.