Memory usage when syncing 1TB with 700k+ files

Just would like to understand what kind of memory usage to expect when syncing 1TB with 700k+ files. The initial folders at both machines where almost identical and the 50Gbyte that needed to be synced is making good progress (syncing is at 40%). I’m using v0.11.17 and syncthing memory usage on the machine that needs to receive files is up to 789MByte. Syncthing on the machine that sends the files only needs 433 MByte memory.

Reading the forum I expected that after the initial scan the memory would drop to 150MByte but its still going up. Or is the memory only going down after the first complete sync?

Oops, syncthing did just crash

Memory is managed by the go runtime, so there is little we can do.

The memory should go down as syncthing becomes idle. While its receiving or sending it obviously has to allocate memory.

You can set GOGC=20 envvar to invoke the GC more often at the cost of CPU.

You can also produce memory profiling info (check -help) for someone to look at if you suspect issues.

Restarted with activating the UI, again a crash Restarted without activating the UI, seems to run fine with about half the memory (400MBytes) Someone interested in the log file of the crash?

Yes, you should just post it, unless its a out of memory crash, in which case a memory profile would be more useful.

New users can’t attach and the log file is too big to include in message Can download from here Sync is at 100% now scanning, left for some time without UI, restarted the UI Now 675MByte memory used.

It looks like a golang runtime bug. I doubt it has anything todo with the fact you have UI enabled or not, and is probably not related to ram. What version of windows are you running?

Judging from the build tag, it seems like it’s the standard build we provide, but I’d like to double check if that is the case?

Also, the log seems interleaved (for if you search for goroutine 94 it actually comes up twice). Can you try running vanilla syncthing, and see if that still happens, and if it does, get a non-interleaved log?

There should also be a log file in syncthing’s home directory which is captured by syncthing (rather than synctrayzor)

Weird,

I’m running the official syncthing 0.11.17 on uptodate windows 7. The log file came from the log folder underneath the home directory of the syncthing executable Would the interleaving indicate I’ve been running syncthing twice? Is that possible? There is something weird as the replication process has produce till now 300000 files conflicting copies in one folder, 8 conflicting copies of each original file in the folder. So I stopped syncing as you can imagine. Will do a restart of the syncing tomorrow as the replication target is being used for more important stuff (watching TV series)

So there is something really weird going on. So just to get things straight, you do seem to be using synctrayzor by @canton7. If the file you’ve provided is you think syncthings log, then I have to tell you that it is not (as it got a few of Syntrayzor’s log lines).

Syncthing by itself also generates a log on windows (if you use it without synctrayzor), which is the one I am interested in. Alternatively, you can invoke syncthing by itself, enable a fair share of scroll-back buffer in the command prompt, and try triggering the crash (so I could submit it to golang devs).

Alternatively, if you open up command prompt, and check -help for syncthing, it will tell you it’s home directory, which is where the log will be (make sure it’s not the database log, but the same log you get in the command prompts buffer).

Regarding the conflicts, it could happen if you zapped the database between the syncs, or if you have some funky backing storage (samba sometimes does it) which does not preserve modification timestamps.

Will run syncthing from a clean command window tomorrow without synctrayzor to test if works better.

After the first and only crash it did not crash anymore

Backing store is on both machines on local ssd/hdd

That unexpected signal during runtime execution stuff is a Go bug, for what it’s worth. It should be filed, if it isn’t already… Although a reproducer will probably be necessary to get it fixed.

Most of these seem to be closed in Go 1.5, some related to float conversions, some related to GC, some related to reflections package.

Yep that’s a SyncTrayzor log. Syncthing’s raw log is in C:\Users\<You>\AppData\Local\Syncthing\syncthing.log.

I should probably stop keeping a separate log - there’s no point in it…

The worse part is that its somehow interleaved.

Using plain syncthing everything did run well. Total memory used peaked at 780MBytes, went down to about 150MBytes after 30 minutes Topic can be closed (unless someone comes with a solution for substantial less memory usage)

SyncTrazyor won’t affect the memory that Syncthing uses (very much) - the two run as independent processes. If Syncthing is now using less memory on its own, it will use less memory when running inside SyncTrayzor.

Well querying stuff might have the effect he’s seeing. But it’s our fault, not yours.

I tested with synctrayzor, syncthing-gtk and plain syncthing/syncthing-inotify and behavior is similar. No need to investigate further.