Syncthing killed repeatedly - no errors

Sounds reasonable.

I actually had a similar problem with the 32bit dev-build (from another ticket) on my windows machine this morning. There was a popup from windows saying syncthing had a memory problem. Didn’t think about it more though. But synchting was using almost 1GB of RAM at that time.

Two questions:

  • Could you add a proper error-message in that case so it’s easy to understand?
  • Is there any way to work around this? I mean the machine is 32bit with low RAM, can’t change that. Maybe could you optimize / reduce the memory-usage?

If it dies due to failure to allocate there is a very verbose panic message, I just it just wasn’t captured by systemd or you didn’t see it. If it gets killed by Linux due to OOM, there is no chance to emit a message. The dmesg log will have a message from the kernel about it.

Generally speaking it’s fairly optimized as is. The known exceptions are very large files (which soon will be better with variable block size) and large folders where a lot has changed (because all of the changed files get queued).

1 Like

Yep you were right: (from dmesg log)

[998745.207835] Out of memory: Kill process 21850 (syncthing) score 826 or sacrifice child
[998745.207975] Killed process 21850 (syncthing) total-vm:1340936kB, anon-rss:856940kB, file-rss:0kB, shmem-rss:0k

When is soon? But anyway i think in my case it’s a lot of small files, a few medium sized and no huge files.

So the problem must be related to the second part:

Could this be done with the help of a temp file? Maybe just if memory runs low?

Databases do this all the time, when sorting stuff that does not fit in memory, they use temp files on disk.

Set scanProgressInterval to a negative value in the config and see if that helps, or enable swap.

I tried enabling swap. I increased it two times. My final setup is this:

  • RAM: 1G
  • SWAP: 2G

Result:

  • Syncthing crashed again:

I have the feeling that the last update degenerated the needed memory for a scan.

Before it was at least able to download what it got here.

(Not sure though. Could also be that it never had to do a full scan yet, while just downloading stuff …)

I will try again with more swap, but not sure if the system supports that.

But maybe you can take a look again on your side, too.

Greetings Fred;

What does this do?

Disables the cache which allows to calculates scan progress, so your scans won’t have progress but they won’t consume as much memory.

There are a few other settings that could reduce memory usage, yet I suspect it shouldn’t be this high to start with. Can you try redownloading the binary to make sure it’s not corrupt?

I can not further increase swap. It just won’t work. Ill’ try the scanProgressInterval setting next.

Did you try redownloading the binary? Can you post your config?

I don’t think that could / should be a problem. Did this ever happen?
I use apt-get to install it. How should it be corrupted without being totally broken?

Here it is (anonymized): config.xml (41.4 KB)

Ok i did this and restarted the scan NOW (21:35 Uhr German time).
So let’s see what happens.

Currently at: 118 MB RAM

Last scan: 2018-04-10 20:33:25 (which did crash)

Update: 22:19 Uhr German time
So far the scan is still running and the RAM usage did only increase slightly to: 129 MiB

Disk I/O for read and writes varies between 0.2 and 4MB/s.

I’ll let you know tomorrow if it was successful.

Yes, multiple times

The scan was yet again not successful.

But looking in dmesg it does not seem like a memory related problem this time:

[Mi Apr 11 18:05:54 2018] systemd[1]: Starting Syncthing - Open Source Continuous File Synchronization for sync...
[Mi Apr 11 18:05:54 2018] systemd[1]: Started Syncthing - Open Source Continuous File Synchronization for sync.
[Mi Apr 11 18:08:10 2018] systemd[1]: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 7551 (du)
[Mi Apr 11 18:08:10 2018] systemd[1]: Mounting Arbitrary Executable File Formats File System...
[Mi Apr 11 18:08:10 2018] systemd[1]: Mounted Arbitrary Executable File Formats File System.
[Mi Apr 11 19:10:26 2018] systemd[1]: Starting Session c11 of user pi.
[Mi Apr 11 19:10:26 2018] systemd[1]: Started Session c11 of user pi.
[Mi Apr 11 19:10:39 2018] systemd[1]: Starting Session c12 of user pi.
[Mi Apr 11 19:10:39 2018] systemd[1]: Started Session c12 of user pi.
[Mi Apr 11 19:13:09 2018] systemd[1]: Starting Session c13 of user pi.
[Mi Apr 11 19:13:09 2018] systemd[1]: Started Session c13 of user pi.
[Mi Apr 11 21:11:22 2018] systemd[1]: syncthing@sync.service: main process exited, code=exited, status=2/INVALIDARGUMENT
[Mi Apr 11 21:11:22 2018] systemd[1]: Unit syncthing@sync.service entered failed state.
[Mi Apr 11 21:11:22 2018] systemd[1]: syncthing@sync.service holdoff time over, scheduling restart.
[Mi Apr 11 21:11:22 2018] systemd[1]: Cannot add dependency job for unit syncthing-inotify@sync.service, ignoring: Unit syncthing-inotify@sync.service failed to load: No such file or directory.
[Mi Apr 11 21:11:22 2018] systemd[1]: Stopping Syncthing - Open Source Continuous File Synchronization for sync...
[Mi Apr 11 21:11:22 2018] systemd[1]: Starting Syncthing - Open Source Continuous File Synchronization for sync...
[Mi Apr 11 21:11:22 2018] systemd[1]: Started Syncthing - Open Source Continuous File Synchronization for sync.
[Mi Apr 11 22:43:54 2018] systemd[1]: Stopping User Manager for UID 1000...
[Mi Apr 11 22:43:57 2018] systemd[1]: Stopped User Manager for UID 1000.
[Mi Apr 11 22:43:57 2018] systemd[1]: Stopping user-1000.slice.
[Mi Apr 11 22:43:57 2018] systemd[1]: Removed slice user-1000.slice.
[Do Apr 12 00:38:42 2018] systemd[1]: Starting Cleanup of Temporary Directories...
[Do Apr 12 00:38:48 2018] systemd[1]: Started Cleanup of Temporary Directories.
[Do Apr 12 01:12:51 2018] systemd[1]: syncthing@sync.service: main process exited, code=exited, status=2/INVALIDARGUMENT
[Do Apr 12 01:12:51 2018] systemd[1]: Unit syncthing@sync.service entered failed state.
[Do Apr 12 01:12:51 2018] systemd[1]: syncthing@sync.service holdoff time over, scheduling restart.
[Do Apr 12 01:12:51 2018] systemd[1]: Cannot add dependency job for unit syncthing-inotify@sync.service, ignoring: Unit syncthing-inotify@sync.service failed to load: No such file or directory.
[Do Apr 12 01:12:51 2018] systemd[1]: Stopping Syncthing - Open Source Continuous File Synchronization for sync...
[Do Apr 12 01:12:51 2018] systemd[1]: Starting Syncthing - Open Source Continuous File Synchronization for sync...
[Do Apr 12 01:12:51 2018] systemd[1]: Started Syncthing - Open Source Continuous File Synchronization for sync.
[Do Apr 12 05:26:36 2018] systemd[1]: syncthing@sync.service: main process exited, code=exited, status=2/INVALIDARGUMENT
[Do Apr 12 05:26:36 2018] systemd[1]: Unit syncthing@sync.service entered failed state.
[Do Apr 12 05:26:36 2018] systemd[1]: syncthing@sync.service holdoff time over, scheduling restart.
[Do Apr 12 05:26:36 2018] systemd[1]: Cannot add dependency job for unit syncthing-inotify@sync.service, ignoring: Unit syncthing-inotify@sync.service failed to load: No such file or directory.
[Do Apr 12 05:26:36 2018] systemd[1]: Stopping Syncthing - Open Source Continuous File Synchronization for sync...
[Do Apr 12 05:26:36 2018] systemd[1]: Starting Syncthing - Open Source Continuous File Synchronization for sync...
[Do Apr 12 05:26:36 2018] systemd[1]: Started Syncthing - Open Source Continuous File Synchronization for sync.

Apparently the scan completed now. The folder is syncing now (after i rebooted the whole device).

Thanks for the help!

Would it make sense to use a disk-spilling queue in both scanner and puller? Scanner is a bit more problematic as we store a file info (without block though), not just file name. It seems pretty doable to use an adjusted version of the index sorter for this. It’s not the nicest solution considering the disk io discussions going on, but better than OOM crashes and short of walking the folders twice, I don’t see a possibility to have progress updates without intermediately storing files to be processed. I’d definitely propose to use some heuristic criterion for spilling, not a fixed max size, to prevent spilling on systems that can take the memory spike.

Sorry, i’m back again …

Now i have OOM problems again, but this time while syncing (the scan completes ok with the new settings):

SyncthingCrashedAgain.txt (193.2 KB)

This happened several times now. But this time in noticed it in the logs.

The memory usage rises up to at least 1.9G (displayed in the UI).

I will pause the big folder now:
It’s a little bit above 1 million files and about 78GB in size with about 30k folders.

+1 for “soon” :slight_smile:

It is sorely needed…

I had to move Syncthing from my NAS box (4GB) to the main PC (16GB) to get it working. And now of course watching for changes does not work (on CIFS mounts)

Thanks!

We could do two scans, one to figure out the amount of data that needs to be hashed, and once more to do it. We might run into a discrepancy and need to handle that fact of course.

Disk-spilling for this purpose does not sound worth it to me.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.