Excessive RAM Usage - Continued

Hello:

Continuing from Excessive RAM Usage

I’m still investigating this, and will be rebuilding shortly - but it does appear I’m still getting high RAM usage.

I’m currently running 0.14.49rc1 with https://github.com/calmh/syncthing/commit/82c5fc008ce82a02b5d5ad1bab124d4648dca724 patched over it. Because of a deadlock issue discussed here - Potential Deadlock Issue on Custom Build - I’ll shortly be rebuilding against master with that same patch applied - but I’m still seeing high RAM usage on my system:

No, obviously, I do have a lot of files - and there is a lot of syncing still going on - but I’d be grateful to know if there are any other measures I can take to reduce my memory footprint.

I’ve grabbed a couple of heap profiles - are they safe to upload publicly without exposing private data?

Thanks,

Pants.

Yep, please upload.

Thanks Jakob. Here they are:

syncthing-heap-linux-amd64-v0.14.49-rc.1 6-gd443c254-094131.pprof (528.2 KB)

syncthing-heap-linux-amd64-v0.14.49-rc.1 6-gd443c254-094346.pprof (563.4 KB)

Just a tip - whatever you do, don’t split your one massive folder into lots of small folders. It makes things worse.

I have ~150gb of backups that I’m syncing around spread over 36 folders, and when you have multiple folders on the same slow disk, SyncThing scans all the folders simultaneously, resulting in long disk queues and a complete inability to get anything done.

See: Disk Usage

Also, this isn’t just my development machine. If anything goes wrong on the production database server, my first step is to stop SyncThing because 90% of the problems I’ve had with it have been traced back to SyncThing either using all the disk bandwidth or all the CPU or all the memory. And then of course, when I restart SyncThing it does a scan. So I have to restart SyncThing, pause all the folders, and then restart them one at a time to get it back on its feet.

I was actually considering changing the configuration to “one huge folder” rather than “lots of big folders” but I guess I won’t do that now.

I just want SyncThing to run in the background moving databases off the database servers and builds onto the web servers and to never get in the way of any of my foreground activities. I don’t want it to slow down either my production database server or my development machine whilst it does its thing. I don’t care if it takes twice as long to sync, as long as it does so reliably, and without affecting performance.

[Note - of course on my production server the backups are on a different physical drive to the databases and the logs and the system so SyncThing causing massive disk queues shouldn’t affect performance. But it does. Not sure how.]

A significant chunk is ignore caching. You should turn that off (it’s off by default as well). Another big chunk is file metadata while scanning. I’m not sure why that is, will look into it.

Can you post your config? The whole thing looks a bit odd to me, you may be running into some corner/edge case we haven’t thought of.

Thanks Jakob.

Curious - I’m certain I didn’t manually enable that.

Sure - here’s a sanitised version for you. Let me know if you need me to privately send the raw version.

config-sanitised.xml (14.4 KB)

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.