Raspberry Pi3 keeps crashing while syncing.

Support
1

Hi all,

Since Syncthing upgrade, both of my raspberry Pi3 keeps crashing due to Out of memory problems. (Dmesg dump: https://pastebin.com/gYUvxyK4). One of the RPI’s is dedicated to Syncthing only.

How can I restrict memory use of Syncthing?

I’m starting Syncthing from systemd (systemctl) with following parameters. [Unit] Description=Syncthing - Open Source Continuous File Synchronization for %I Documentation=man:syncthing(1) After=network.target

[Service]
User=%i
Nice=19
Environment=GOMAXPROCS=1
ExecStart=/usr/bin/syncthing -no-browser -no-restart -logflags=0
Restart=on-failure
SuccessExitStatus=3 4
RestartForceExitStatus=3 4

# Hardening
ProtectSystem=full
PrivateTmp=true
SystemCallArchitectures=native
MemoryDenyWriteExecute=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

Hope someone can shed a light.

2

Could it be the problem suggested in post:https://forum.syncthing.net/t/increased-memory-usage-since-0-11-2/2690/20

If so, what BlockCache is recommended for a RP3?

... 0
3

Syncthing logs before the crash might be more helpful. You can set progressUpdateIntervalS and scanProgressIntervalS to -1 to reduce the memory footprint. Which version are you on?

4

Hi, Version is 1.1.4.

I’ve set databaseBlockCacheMiB to 1 and i’m trying again.

I will set progressUpdateIntervalS and scanProgressIntervalS to -1. What does it do?

5

I don’t think there’s such a thing as databaseBlockCacheMiB, and I’d have no idea what it would be about.

It won’t track files that are scanned/synced, i.e. less memory required but you won’t get a progress percentage in the UI.

6

No difference setting progressUpdateIntervalS.

databaseBlockCacheMiB is a setting Jakob Borg suggested in another post, about similar problem.

Nothing in syncthing logs: https://pastebin.com/XPVf67dT

both RPi3 keeps crashing… going back to 0.14.18, see what happens…

7

You are looking at the wrong logs, this is the database log.

8

I see. It is in syslog. Ok, final try, then I will wait for 1.2.0.

Anybody got experience with 1.14 on RPi with large datasets ( Files:555.166 Dirs 124.745 Total size 3,02 TiB)?

9

Again a freeze. Nothing in syslog/Messages, only in dmesg.

Any suggestions?

> [ 6633.461453] INFO: task kworker/2:2:2161 blocked for more than 120 seconds.
> [ 6633.461468]       Tainted: G         C        4.19.42-v7+ #1219
> [ 6633.461473] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [ 6633.461481] kworker/2:2     D    0  2161      2 0x00000000
> [ 6633.461517] Workqueue: events_freezable mmc_rescan
> [ 6633.461549] [<8082642c>] (__schedule) from [<80826a9c>] (schedule+0x50/0xa8)
> [ 6633.461567] [<80826a9c>] (schedule) from [<8067cc9c>] (__mmc_claim_host+0x120/0x228)
> [ 6633.461586] [<8067cc9c>] (__mmc_claim_host) from [<8067cddc>] (mmc_get_card+0x38/0x3c)
> [ 6633.461604] [<8067cddc>] (mmc_get_card) from [<80685e50>] (mmc_sd_detect+0x24/0x7c)
> [ 6633.461623] [<80685e50>] (mmc_sd_detect) from [<8067f404>] (mmc_rescan+0x1cc/0x39c)
> [ 6633.461642] [<8067f404>] (mmc_rescan) from [<8013bd7c>] (process_one_work+0x170/0x458)
> [ 6633.461660] [<8013bd7c>] (process_one_work) from [<8013c0c0>] (worker_thread+0x5c/0x5a4)
> [ 6633.461678] [<8013c0c0>] (worker_thread) from [<801423ac>] (kthread+0x138/0x168)
> [ 6633.461698] [<801423ac>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28)
> [ 6633.461706] Exception stack(0x8dbc7fb0 to 0x8dbc7ff8)
> [ 6633.461717] 7fa0:                                     00000000 00000000 00000000 00000000
> [ 6633.461729] 7fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[ 6633.461740] 7fe0: 00000000 00000000 00000000 00000000 00000013 00000000

dome

10

Seems like the SD card is giving up. SD cards do have a lifetime, so keeping things like syncthings database on it will probably kill it early.

11

Hi,

Thank you for quick answer! To prefent this, I moved database to external drive. Also have same problem on another RPi3. Pulling my hair out over this. (if i had any…)

12

Waiting for 1.2.0 won’t help. If the OOM occurs during pulling, it might be a known problem (pull queue can get huge and there’s no mechanism preventing that).

To debug this a good option to prevent flooding your system logs is to run Syncthinglogging, not through systemd, and enable model debug logging (e.g. STTRACE=model syncthing -no-browser -no-restart 2>&1 | tee syncthing.log). Then if you get something with *folder label* (*folder id*) pulling, then a ton of lines about specific files, and finally kill by OOM - pulling is indeed the problem and it would be very nice if you could share the logs (privately if the filenames are sensitive). There’s an ongoing change proposal that addresses this issue (pull queue growing too large) and seeing which part of the queue is getting flooded might give some clues for that.

13

I will send you the debug logs.

Now I can only hope the pull-queue problem is getting fixed.

14

For now i’m bypassing the big pull queue problem by using the “ignoring patterns” and gracefully deleting a filetype in the list of patterns on the receiving side each sync. It’s slow but works for now. (+3TB Send Only folder on NAS, 2 RPi3’s with external powered USB drives in Receive Only mode)

15

Or add some swap?

16

mmm… swap on an external drive, wouldn’t the kernel panic if the drive is not mounted after reboot?

17

sorry I have no idea if it would or not :frowning: but could you use a spare flash drive you might have floating around for the swap memory to reduce the chances of it being removed by a user? or is the concern that the drive might not be mounted in the correct order at boot time?

18

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.