Initial sync still deathly slow on the RPi

Support
1

Well, it seems I’m a victim of my own success.

For the full saga, see here and here. Although the current issue involves the same project and devices, this is a new problem and so calls for a new topic.

After much angst I finally got the initial sync to kick off and run. It did great for a bit, but then at about the 20% mark or so it ground to a screeching halt. The Web GUI became unresponsive, as did my VNC connection. The only fix was to pull the plug and power cycle the RPi. Ouch. One never likes to do that.

And so began the cycle of pain. After the restart I was good for about half a minute, frantically issuing commands at the terminal while I could, and then it’d lock up again. I tried setting the GOMAXPROCS environment variable to 2, as indicated in the docs, and that helped a little bit. I was now able to get a full minute in before the lock-up.

Setting it to 1 has proven to be the most reliable, as I can at least get a few minutes at a time in at the terminal. But then the service crashes (consistently). It got so bad that I ended up creating a second service to keep an eye on Syncthing and restart it after it crashes. Here’s some recent log output from that one:

Thu 16 Feb 2023 12:54:38 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 01:29:16 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 01:36:53 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 01:59:04 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 02:33:01 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 02:41:15 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 02:50:55 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 02:53:05 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 03:00:21 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 03:18:07 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 03:26:04 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 03:34:26 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 03:42:27 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 03:50:30 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 03:58:44 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 04:06:33 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 04:14:21 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 04:22:27 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 04:30:21 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 04:38:12 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 04:46:25 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 04:54:14 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 05:02:10 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 05:09:56 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 05:17:27 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 05:25:39 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 05:28:00 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 05:34:59 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 05:43:00 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 05:45:21 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 05:52:57 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 05:55:14 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 06:02:31 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 06:10:23 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 06:18:19 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 06:26:36 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 06:34:42 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 08:30:57 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 08:39:25 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 10:07:52 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 10:16:24 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 10:24:34 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 11:07:16 AM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 12:37:57 PM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 12:47:05 PM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 12:57:43 PM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 01:06:14 PM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 01:14:59 PM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 01:23:39 PM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 01:32:16 PM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 01:40:45 PM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 02:55:13 PM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 03:03:52 PM AKST - syncthing@admin.service restarted
Thu 16 Feb 2023 03:12:38 PM AKST - syncthing@admin.service restarted

As we can see, there’s no discernable time-based pattern. It scans for a bit, syncs a few files, and then crashes again. The only thing consistent about it all is the sequence of events:

SCAN -> SYNC -> CRASH — SCAN -> SYNC -> CRASH

I’m about a third of the way through, with each cycle trickling in a few new files. I’m on the edge here, debating whether I should pause the folder (or even remove it) and go back to running the initial sync with rsync. At least then the CPU would only be tied up for the SCAN phase.

Again, I’m at a loss. Getting this going on the RPi has proven troublesome—the other devices synced right up at the time. There may be additional configurations to throttle CPU time, I don’t know.

Any suggestions for improvement are welcome.

image
image1373×1049 268 KB

2

hmmm… VNC … that means a desktop environment on the RPi.

Although not a deal breaker, an OS + DE + VNC + Syncthing (with a few hundred thousand files) all in under 1GB of RAM is living on the edge. :wink:

Personally, I’d go headless on the RPi to free up system resources. Set Linux to boot into runlevel 3 instead of 5 and either enable external access to Syncthing’s web GUI or use a SSH tunnel to port forward connections from a local web browser.

Does your RPi boot via a SD card, USB flash drive or PXE?

If it’s a SD card or USB flash drive, check the media for errors. With a few hundred thousand files, Syncthing is going to be doing a lot of writing to its database files.

1 Like
3

Aha. Now we’re getting somewhere.

OK, I’m game! I’ve already enabled external access to the Web GUI, so that’s covered. You’re saying something about something called a runlevel 3? Will that tell it to boot without a desktop? I guess I’ll have to do some RTFM.

That’s the one. Although PXE doesn’t sound like a bad idea. I’ll have to look into that. Avails of more horsepower, yes?

I imagine I’ll have to stand up my RPi4 for that one, wouldn’t you agree? I’m not sure I’m comfortable doing that on a Windows OS.

4

image

I think I’ve reached Level 3 :wink:

5

Goodness, @gadget… that has made all the difference in the world!

I didn’t even have to back off my GOMAXPROCS throttle, and it’s tossing around files at a tremendous rate.

I do believe you’ve saved the day :smiley: :smiley: :smiley:

6

PXE can be leaner, but it depends on the particular setup. Since PXE requires a DHCP server, a TFTP server, and often a file server (can be separate hosts or all-in-one), it might not be worth the effort for a single RPi.

There are quite a few flash media diagnostic tools available for Windows so it’s certainly an option.

I normally use badblocks and check the output from dmesg for errors.

7

I arrived at that very conclusion after looking into it only briefly.

OK, thanks.

Again.

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.