Syncthing crashing during initial scan

Support
1

I’m running syncthing on a Raspberry Pi Zero W and trying to sync a 50gb folder on an external SATA drive connected over USB. After a while the process will halt and the drive stops seeing any activity. There is no logs over journalctl and nothing of note happens over syncthing logs:

● syncthing@syncthing.service - Syncthing - Open Source Continuous File Synchronization for syncthing
     Loaded: loaded (/lib/systemd/system/syncthing@.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-08-03 11:19:12 BST; 26min ago
       Docs: man:syncthing(1)
   Main PID: 389 (syncthing)
      Tasks: 16 (limit: 414)
        CPU: 18min 18.239s
     CGroup: /system.slice/system-syncthing.slice/syncthing@syncthing.service
             ├─389 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0
             └─405 /usr/bin/syncthing serve --no-browser --no-restart --logflags=0

Aug 03 11:23:14 raspberrypi syncthing[389]: [G2JAE] INFO: Device 72P353T-7GBTOTW-3NLHFFM-YD4545X-RGJYKEW-YTDPR3K-R6LRTZP-ZRRFRAA >
Aug 03 11:24:09 raspberrypi syncthing[389]: [G2JAE] INFO: Completed initial scan of sendreceive folder "dl-server/primary" (d5har>
Aug 03 11:24:09 raspberrypi syncthing[389]: [G2JAE] INFO: Completed initial scan of sendreceive folder "dl-server/management" (dc>
Aug 03 11:24:24 raspberrypi syncthing[389]: [G2JAE] INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
Aug 03 11:24:34 raspberrypi syncthing[389]: [G2JAE] INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) shutting>
Aug 03 11:24:34 raspberrypi syncthing[389]: [G2JAE] INFO: listenerSupervisor@dynamic+https://relays.syncthing.net/endpoint: servi>
Aug 03 11:24:34 raspberrypi syncthing[389]: [G2JAE] INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
Aug 03 11:25:28 raspberrypi syncthing[389]: [G2JAE] INFO: Joined relay relay://148.251.237.21:22067
Aug 03 11:28:16 raspberrypi syncthing[389]: [G2JAE] INFO: quic://0.0.0.0:22000 detected NAT type: Port restricted NAT
Aug 03 11:28:16 raspberrypi syncthing[389]: [G2JAE] INFO: quic://0.0.0.0:22000 resolved external address quic://91.224.192.163:22>
lines 1-21/21 (END)

I can tell there’s nothing happening because the scan isn’t even nearly done and the drive activity indicator has stopped flashing. I also can’t access the GUI.

An error message appears in the console:

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.714222] Internal error: Oops: 8000000d [#1] ARM

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715361] Process syncthing (pid: 482, stack limit = 0x9c8a9c36)

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715392] Stack: (0xdc90ddc0 to 0xdc90e000)

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715423] ddc0: c0df7444 c01f09d8 c1f07d70 dc90de1f 0f140003 fbff43bd 00000001 dc90deec

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715457] dde0: 00000004 c6ce4dc8 c0c2f02c dc90de58 00000000 dc90de1f c0cb92d4 c01f1218

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715490] de00: dc90de1f 00000000 c0c2f7dc 00100cca 00025b92 cbba2ea4 00000001 01439340

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715522] de20: 00000000 c19ab5e0 00000000 00010001 c3000100 dc90de34 dc90de34 00000008

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715555] de40: 00000008 012dc900 012dc880 012dc800 012dc780 012dc700 012dc680 012dc600

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715588] de60: 012dc580 fbff43bd c6ce4dc8 00000000 dc90deec 00025b92 00000040 c0c2f02c

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715620] de80: c6ce4dc8 00000000 c2f6ee40 c01cd7e0 c3073cd8 cba0c554 c5f68380 00000a55

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715652] dea0: cbaea368 00000255 14863000 c6ce4dc8 00000040 c0c2f02c dc90dfb0 00000002

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715685] dec0: c5f683b0 c01ce42c 00000000 fbff43bd 00000000 c6e13320 0000000f 0000000f

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715718] dee0: c558b900 c5f68380 ffffffff c6ce4dc8 00000cc0 00014863 14863000 14863000

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715750] df00: 00000a55 c5e78520 c5e78520 012dc900 00000000 00000000 00000000 00000000

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715781] df20: 00000000 fbff43bd 00000817 dc90dfb0 c5f68380 14863000 00000817 00000255

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715815] df40: c6ce4dc8 00000002 c5f683b0 c08cd358 00000000 c6e13323 c6e13320 c0c2f02c

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715847] df60: 08e3f500 c0c34bfc 00000817 c08cd220 14863000 dc90dfb0 00000000 02802f00

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715879] df80: 14880000 c00189fc 00000000 00000000 02802f48 c0008404 0008be0c 80000010

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715911] dfa0: ffffffff c6d46180 00c5387d c0008f94 00000000 00000000 00000000 00000000

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715943] dfc0: 00000000 00000000 00000000 00000000 14863000 00000000 02802f00 14880000

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.715975] dfe0: 1487ffe1 0ede3870 0001d5fc 0008be0c 80000010 ffffffff 00000000 00000000

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.716014]  swap_readpage from swapin_readahead+0x344/0x42c

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.716066]  swapin_readahead from do_swap_page+0x41c/0x7f8

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.716129]  do_swap_page from handle_mm_fault+0x5b0/0xd94

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.716177]  handle_mm_fault from do_page_fault+0x138/0x3a8

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.716239]  do_page_fault from do_DataAbort+0x40/0xb4

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.716299]  do_DataAbort from __dabt_usr+0x54/0x60

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.716341] Exception stack(0xdc90dfb0 to 0xdc90dff8)

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.716368] dfa0:                                     00000000 00000000 00000000 00000000

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.716401] dfc0: 00000000 00000000 00000000 00000000 14863000 00000000 02802f00 14880000

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.716433] dfe0: 1487ffe1 0ede3870 0001d5fc 0008be0c 80000010 ffffffff

Message from syslogd@raspberrypi at Aug  3 11:44:59 ...
 kernel:[ 1371.716463] Code: 00000000 00000000 00000000 00000000 (00000000)

Attempts to restart the system service take a long time and don’t work:

blackshibe@raspberrypi:~ $ sudo systemctl restart syncthing@syncthing.service
blackshibe@raspberrypi:~ $ sudo systemctl status syncthing@syncthing.service
● syncthing@syncthing.service - Syncthing - Open Source Continuous File Synchronization for syncthing
     Loaded: loaded (/lib/systemd/system/syncthing@.service; enabled; vendor preset: enabled)
     Active: deactivating (final-sigkill) (Result: exit-code) since Thu 2023-08-03 11:50:49 BST; 1min 56s ago
       Docs: man:syncthing(1)
    Process: 810 ExecStart=/usr/bin/syncthing serve --no-browser --no-restart --logflags=0 (code=exited, status=1/FAILURE)
   Main PID: 810 (code=exited, status=1/FAILURE)
      Tasks: 2 (limit: 414)
        CPU: 1.153s
     CGroup: /system.slice/system-syncthing.slice/syncthing@syncthing.service

Aug 03 11:50:48 raspberrypi systemd[1]: Started Syncthing - Open Source Continuous File Synchronization for syncthing.
Aug 03 11:50:48 raspberrypi syncthing[810]: [start] INFO: syncthing v1.23.6 "Fermium Flea" (go1.20.5 linux-arm) debian@github.syncthing.net 2023-07-03 09:51:12 UTC [noupgrade]
Aug 03 11:50:49 raspberrypi syncthing[810]: [start] WARNING: Error opening database: resource temporarily unavailable (is another instance of Syncthing running?)
Aug 03 11:50:49 raspberrypi systemd[1]: syncthing@syncthing.service: Main process exited, code=exited, status=1/FAILURE
Aug 03 11:50:49 raspberrypi systemd[1]: syncthing@syncthing.service: Killing process 406 (syncthing) with signal SIGKILL.
Aug 03 11:52:19 raspberrypi systemd[1]: syncthing@syncthing.service: Processes still around after SIGKILL. Ignoring.
Aug 03 11:52:19 raspberrypi systemd[1]: syncthing@syncthing.service: Killing process 406 (syncthing) with signal SIGKILL.
2

I don’t see a Syncthing crash here, I see a kernel oops during swapping, caused by probably either a kernel bug or (more likely) a hardware problem, in turn possibly caused by overheating or something similar.

1 Like
3

It does seem to have been overheating, even though I had a passive cooling case mounted. I wiped the drive and decided to sync the folders from scratch instead of letting syncthing rummage through an already ready to sync drive; It’s been slowly but steadily downloading everything without issue so far.

For my use case this tiny little thing is most likely not going to be enough. It’s not even half done after an entire day of waiting. Shame.

4

The Pi Zero’s 512MB of RAM isn’t much to work with for an application such as Syncthing. There’s likely a whole lot of memory swapping happening.

Could you post the output from dmesg -T and free?

Which Linux distribution are you using?

How is the SATA drive formatted?

1 Like
5

dmesg output: dmesg.txt (30.4 KB)

free output:

               total        used        free      shared  buff/cache   available
Mem:          439592      120944       84420         172      234228      260536
Swap:        1150968       36864     1114104

^ I configured another 1GB swapfile to try to solve the freezing issue.

I’m using Raspbian Lite:

blackshibe@raspberrypi:~ $ neofetch
  `.::///+:/-.        --///+//-:``    blackshibe@raspberrypi
 `+oooooooooooo:   `+oooooooooooo:    ----------------------
  /oooo++//ooooo:  ooooo+//+ooooo.    OS: Raspbian GNU/Linux 11 (bullseye) armv6l
  `+ooooooo:-:oo-  +o+::/ooooooo:     Host: Raspberry Pi Zero W Rev 1.1
   `:oooooooo+``    `.oooooooo+-      Kernel: 6.1.21+
     `:++ooo/.        :+ooo+/.`       Uptime: 59 mins
        ...`  `.----.` ``..           Packages: 824 (dpkg)
     .::::-``:::::::::.`-:::-`        Shell: bash 5.1.4
    -:::-`   .:::::::-`  `-:::-       Terminal: /dev/pts/0
   `::.  `.--.`  `` `.---.``.::`      CPU: BCM2835 (1) @ 1.000GHz
       .::::::::`  -::::::::` `       Memory: 115MiB / 429MiB
 .::` .:::::::::- `::::::::::``::.
-:::` ::::::::::.  ::::::::::.`:::-
::::  -::::::::.   `-::::::::  ::::
-::-   .-:::-.``....``.-::-.   -::-
 .. ``       .::::::::.     `..`..
   -:::-`   -::::::::::`  .:::::`
   :::::::` -::::::::::` :::::::.
   .:::::::  -::::::::. ::::::::
    `-:::::`   ..--.`   ::::::.
      `...`  `...--..`  `...`
            .::::::::::
             `.-::::-`

the SATA drive is a 256GB SSD partitioned to have a single ext4 volume connected over a USB adapter. My laptop could write data to it at roughly 100mb/s so I didn’t think much else of it.

I should note this is part of a hasty migration away from AWS to avoid paying their storage fees. I was previously hosting about 150GB data there and thought a small SBC might be enough for the occasional file uploads since I don’t usually send over gigabytes of data, more several 10-30mb file uploads a day, plus the files aren’t ever locally altered. I wasn’t expecting the initial sync to be such a headache and I’m considering refunding this to replace it with something like a Banana Pi BPI-M2.

6

Kernel pages cannot be swapped out, leaving ~320MB for everything else that can be.

For comparison, Syncthing on my home server is currently handling ~2,200 directories with ~11,000 files (~2GB total) while averaging ~105MB of physical memory.

Definitely lightweight compared to the full edition, but even sans-DE, it needs extra tuning to run an app like Syncthing with only ~320MB.

Some suggestions that should help (keeping in mind it’s still a RPi Zero :wink:) …

zram

Raspbian includes Linux’s zram kernel module (easiest way to check is with the command modinfo zram).

To check if it’s loaded, lsmod | grep zram (your dmesg -T log output showed it wasn’t being used)

If it’s available, configure it to load at boot via /etc/modules-load.d.

A dmesg sample snippet from my home server:

[Fri Jul 28 14:24:49 2023] zram: Added device: zram0
[Fri Jul 28 14:24:50 2023] systemd[1]: Created slice system-systemd\x2dzram\x2dsetup.slice - Slice /system/systemd-zram-setup.
[Fri Jul 28 14:24:50 2023] zram0: detected capacity change from 0 to 11845632
[Fri Jul 28 14:24:51 2023] Adding 5922812k swap on /dev/zram0.  Priority:100 extents:1 across:5922812k SSDscFS

Modules & Services

Since kernel pages cannot be swapped, review the output from lsmod and disable any unused modules such as for Bluetooth, extra wireless drivers, filesystem modules, etc.

For comparison, my laptop currently loads 172 modules, home file server loads 67, and a micro-server at work loads 42.

Likewise, try to disable as many system services as possible. NetworkManager is great, but a leaner DHCP client and/or static configuration would be better.

For comparison, my laptop is currently running 183 processes with 1,149 threads, home server: 58/82, micro-server at work: 38/247.

Syncthing

See Syncthing’s Configuration Tuning page for options and tips, specifically the Tuning for Low Resources section.

That’s great. I was curious because if it was a NTFS volume there would be more system load (NTFS-3G is FUSE-based, adding overhead due to context switching, memory I/O, bus I/O, etc.). With newer kernels, there’s a native NTFS module for VFS, but it’s usually not used by default.

If your 50GB Syncthing folder has a lot of directories/files, the filesystem watcher will consume more memory (see the inotify(7) manpage).

Since the files are ever locally altered, watching for changes might be unnecessary (expand the Syncthing folder panel → Edit → Advanced → Watch for Changes).

Whether it’ll work better depends on which model of the Banana Pi BPI-M2. If it’s the “Zero”, performance will be worse than the Raspberry Pi Zero W (perhaps even much worse).

Both SBC’s have 512MB of RAM, so no improvement with regards to memory usage.

The BPI-M2 Zero’s “Allwinner H3” CPU is based on an ARM Cortex-A7 core. The Cortex-A7 is 32-bit compared to the Raspberry Pi Zero’s 64-bit Cortex-A53.

(Not to get too nerdy… but the Cortex-A53 is a superscalar processor, which is better for multitasking.)

2 Likes
7

Raspbian includes Linux’s zram kernel module (easiest way to check is with the command modinfo zram).

What makes zram better than a regular swapfile?

For comparison, my laptop currently loads 172 modules, home file server loads 67, and a micro-server at work loads 42.

I removed the camera, bluetooth, video drivers:

blackshibe@raspberrypi:~ $ lsmod
Module                  Size  Used by
8021q                  23069  0
garp                    7841  1 8021q
stp                     2585  1 garp
llc                     6489  2 garp,stp
brcmfmac              299656  0
brcmutil               11424  1 brcmfmac
sha256_generic          4067  0
libsha256               5645  1 sha256_generic
cfg80211              816273  1 brcmfmac
sg                     20596  0
uas                    15055  0
rfkill                 23804  4 cfg80211
raspberrypi_hwmon       2678  0
vc_sm_cma              22121  0
uio_pdrv_genirq         4146  0
uio                    11896  1 uio_pdrv_genirq
fixed                   4791  0
zstd                    3718  1
zstd_compress         348064  1 zstd
zram                   16280  0
zsmalloc               16860  1 zram
ip_tables              13982  0
x_tables               29223  1 ip_tables
ipv6                  494288  32

new RAM usage:

               total        used        free      shared  buff/cache   available
Mem:          439592       86588      177212         824      175792      297320
Swap:         102396           0      102396
Total:        541988       86588      279608

Whether it’ll work better depends on which model of the Banana Pi BPI-M2. If it’s the “Zero”, performance will be worse than the Raspberry Pi Zero W (perhaps even much worse).

The one I was looking at is this one:

image
image998×773 65.9 KB

It has 1GB RAM and a quad-core, which I think makes it far more suitable. I’d also plug it into Ethernet if I could.

8

Since swap space (a file or dedicated partition) is just a collection of O’s and 1’s on storage media, software code contained within it cannot be executed directly by the CPU. Any code and data must first be copied from swap space (aka. “virtual memory”) back into physical memory.

In contrast, zram compresses code and data held in physical memory in the same way that compressing files on a drive saves storage space.

For example, let’s say that you have the following Bash script:

#!/bin/bash

while /bin/true
do
	echo '0000000000'
done

But instead of only a string of ten zeros, there’s one billion of them. Because the script runs in an endless loop, the string of 0’s will eat up 1 GB of RAM.

Although it can be swapped out to virtual memory, the memory pages it occupies have to be swapped back into physical memory to be used.

With zram, the string of one billion zeros ends up as one zero plus a bit of memory overhead for the compression, likely reducing memory usage by at least 90%.

Obviously, the example above is an extreme case, but a lot of programs can be compressed by around 50%. Since zram’s default algorithm is LZO-RLE, I ran a small test with lzop:

$ lzop -o sshd.lzo /sbin/sshd
$ lzop --verbose --list sshd.lzo 
Method         Length    Packed  Ratio     Date    Time   Name
------         ------    ------  -----     ----    ----   ----
LZO1X-1        917192    565944  61.7%  2023-07-19 14:41  sshd

So compressing /sbin/sshd (OpenSSH daemon) reduced it by over 38%.

Syncthing was even better at a 45.5% savings:

$ lzop -o syncthing.lzo /usr/bin/syncthing
$ lzop --verbose --list syncthing.lzo
Method         Length    Packed  Ratio     Date    Time   Name
------         ------    ------  -----     ----    ----   ----
LZO1X-1      24521544  13370165  54.5%  2023-07-30 22:45  syncthing

Of course, zram can be paired with a swap file/partition so it’s not a choice of one or the other. Zram helps reduce swapping to virtual memory. And if the swap file/partition is on flash storage, helps extend the lifespan by reducing the number of write cycles.

If you don’t need firewall and/or IPv6 support, the “ip_tables” and “ipv6” modules can also be unloaded.

The lsmod output doesn’t show any other modules using zstd, so the “zstd” and “zstd_compress” modules can probably also go unless you’re running from a compressed system image, or have some other use for it.

Only time will tell, but based on the example output above, free memory more than doubled from the earlier ~84MB.

Looks like the “BPI-M2 Berry”?

If so, still a step down in horsepower compared to the CPU in the Pi Zero because it’s a ARM Cortex-A7 core (32-bit) vs. a Cortex-A53 (64-bit), but 2x the RAM is an improvement.

3 Likes
9

Looks like the “BPI-M2 Berry”?

If so, still a step down in horsepower compared to the CPU in the Pi Zero because it’s a ARM Cortex-A7 core (32-bit) vs. a Cortex-A53 (64-bit), but 2x the RAM is an improvement.

I opted for the Raspberry Pi 4 model B. It has USB3 support so I get full performance out of the drive.

I considered it because a friend of mine let me ssh into his and the upload benchmarks i did seemed to give me pretty acceptable performance. It was certainly faster at uploading my photos folder (5gb of data)

As for the Raspberry Pi Zero experience in the end…

It’s definitely usable! It syncs files consistently and consumes little power if we skip the outright painful process of getting the files into its local database.

My /etc/modprobe.d/blacklist.conf:

# bluetooth
blacklist bluetooth
blacklist btbcm
blacklist hci_uart
blacklist bnep

# sound
blacklist snd_soc_hdmi_codec
blacklist snd_soc_core
blacklist snd_compress
blacklist snd_pcm
blacklist snd_timer
blacklist snd_bcm2835
blacklist snd

# video / camera
blacklist drm
blacklist vc4
blacklist vc3
blacklist videodev
blacklist fuse
blacklist drm_panel_orientation_quirks
blacklist backlight
blacklist cec
blacklist drm_display_helper
blacklist i2c_bcm2835
blacklist v4l2_mem2mem

blacklist videodev
blacklist videobuf2_v4l2
blacklist videobuf2_vmalloc
blacklist videobuf2_common
blacklist videobuf2_memops
blacklist videobuf2_dma_contig

blacklist bcm2835_isp
blacklist bcm2835_codec
blacklist bcm2835-v4l2
blacklist bcm2835_mmal_vchiq

But It’s not for me. I have roughly a million files to sync and while It consistently scans and uploads files It’s only a few megabytes a second up and down and really slow at accessing the drive in general. Not to mention I can’t just put another microcomputer next to it in case I need to run more than just syncthing 24/7.

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.