Excessive(?) memory usage

I was wondering if someone can shed a bit light on this.

On my workstation I’m running Syncthing (0.14.48, amd64) to synch a fair amount of folders and files to my NAS (0.14.49, aarch64). The synch is essentially one way, all folders on the workstation are configured “Send only”.

My attention was drawn to the memory usage of these programs. According to the respective GUIs, Syncthing on the workstation uses an average of 90MB memory, while Syncthing on the NAS uses almost 1GB. I don’t expect that such a big difference is caused by different hardware or program version. But what then?

Good question. You can take a heap profile and we can answer it.

https://docs.syncthing.net/users/profiling.html

1 Like

I got the heap profiles. Two of each system. Do you have a safe upload address? An can I assume there is no sensitive data in the profile?

There’s nothing private in it, you can attach it here.

syncthing-heap-linux-amd64-v0.14.48-161633.pprof (139.2 KB)

syncthing-heap-linux-arm64-v0.14.49-161558.pprof (121.8 KB)

syncthing-heap-linux-amd64-v0.14.48-161613.pprof (137.4 KB)

syncthing-heap-linux-arm64-v0.14.49-161624.pprof (122.6 KB)

The ARM profile shows a vast amount of memory held by the ignore cache. This is a little bit unexpected. What do your ignore patterns look like, and how many files/directories are in those folders?

A workaround is to disable the ignore caching, trading RAM for CPU while scanning. It’s somewhere among the advanced folder options.

I suspect the !** traverse to infinity and beyond stuff hits the cache big time.

Yes. Then again any sufficiently large folder with a single myveryspecificfile.ext pattern would show the same “cache all the things” behavior.

No, because we don’t traverse down into infinity if there is no exclusions.

Perhaps exclusions = cache disabled.

Or we should move to a bloom filter.

We do if the ignore pattern is just for a specific file or file pattern and doesn’t cut off an entire file tree.

BTW ignore caching defaults to off, as we determined a way back it’s not that advantageous. So I think this is a pathological case probably. You should probably turn it off again, OP.

Anyways, OP should check if ignore caching is enabled, and potentially provide ignore patterns and indication of what the directory tree looks like to better understand this.

1 Like

Ignore caching was disabled on the AMD and enabled on the ARM.

On the ARM, I paused two folders that I suspected to be the cause (containing 1.5M files (files, not bytes) of which maybe 98% were ignored) and the memory dropped (after restart) to 50MiB.

The files are data from dataloggers, 1.360.000, and have names like

data/YYYY/MM/DD/YYYYMMDDHHMMSSx.dat

The entries in the ignore file are like (data logging started in 2015):

data/2015
data/2016
data/2017
data/2018/01
data/2018/02
data/2018/03
data/2018/04
data/2018/05
data/2018/06

If ignore caching means that all ignored file(name)s are cached, I can imagine this is going to cost a lot of memory.

Going by the patterns, I guess files transition from being synced when they are fresh to later becoming ignored as time moves on? If so yes, they would end up in the cache because they are still in the index db. But as noted, ignore caching defaults to off and I think the performance hit of ignores is nowadays quite low even without caching.

Yes, the idea was to release Syncthing from having to repeatedly examining a vast collection of files that wasn’t supposed to be updated anymore.

If the old logs are already on disk on your ARM device, I would remove the ignores, enable fs watcher (if it isn’t already) and set the full rescan intervall to something like 86400 (1 day).

True, but I’m a bit reluctant to put watchers on millions of files that will not change.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.