This is a headless system. I plugged in the keyboard to interact with that ddb prompt, but the keyboard backlight didn’t turn on and the system did not respond to its input. Completely frozen, so I pulled the plug.
On next boot, a manual filesystem check was necessary, as there were inconsistencies to fix. This was expected.
Due to misconfiguration, I don’t have the full syncthing log of that session. Next time it happens, I’ll have it.
Do you think this is a Syncthing problem, or should I be taking this to OpenBSD?
Although Syncthing was mentioned on the console, it was just an unfortunate victim of what looks like a filesystem error: /home: bad dir ino 2436526 at offset 0: mangled entry.
Should run a more extensive media scan of the storage device (microSD?).
USB stick. The root filesystem is on a USB3 stick.
Well, that’s concerning, and I guess there’s nothing I can do to avoid it. It’s a brand new Kingston device, 128GB. Never had any of these failures running Linux, though - because of journalling?
/home is written to heavily over here, and it shows every time a power loss occurs, as it always requires manual intervention with fsck. Many daemons keep their data in a home folder, like /home/_transmission or /home/_syncthing.
It’s not unusual to encounter errors on a brand new USB stick. But note that most USB sticks aren’t designed to be used for running an OS.
The gap has narrowed a lot, but for many years flash memory used in SD cards were of higher quality than in USB sticks because there’s so much demand and competition for USB sticks that the profits are razor thin except for higher capacity and specialized models (e.g. hardware encryption, endurance). Every major manufacturer responded to cheaper brands by adding budget-friendly lines of their USB sticks. Plus with the amount of writing that a typical OS (and Syncthing) needs to do, it wouldn’t be surprising to see bad blocks sooner.
The default filesystem in OpenBSD also supports journaling but it’s been so long since I’ve used OpenBSD that I don’t know how it compares to XFS, ext3/4, btrfs and other filesystems commonly used in Linux. However, I do know that btrfs and some other filesystems in Linux are flash media friendly (i.e. delayed allocation, wear leveling support, etc.)
Filesystem journaling was intended to avoid fsck at boot time by being able to roll back incomplete transactions. So if the filesystem type your USB stick is formatted with is journaling, and a manual fsck is required at boot time, then it’s likely that the USB stick is flaky.
If switching to a different USB stick and/or other changes aren’t an option, the RPi 4 doesn’t need much power, so an inexpensive UPS would be worth it. Even with journaling, checksums and other modern filesystem features, I always plug my servers into a UPS unit.
Searching for journalling on OpenBSD I only found this reddit question from 7 years ago, and the replies unanimously state that there’s no journalling, only “soft updates”, which is disabled by default.
However, I couldn’t find any mention of journalling or soft updates in the FAQ or in related manual pages (fsck, newfs, mount).
I don’t mind doing manual fsck after a power outage. I just wish it didn’t fail while online, as it did.
The UPS and an external SSD are on my wish list. The UPS I was using to power the RPi and all network devices stopped working last week.
I installed OpenBSD 7.4 to see what changed since I last used it. The default filesystem type is FFS2. As of last year, soft updates was changed to a no-op, so whether it’s enabled or not, it does nothing: https://marc.info/?l=openbsd-cvs&m=168856997929968
There has been some work done with journaling in FFS, but so far there doesn’t appear to be any support in OpenBSD.
The downside is that when fsck is required, the damage has already been done. Partially written files are deleted during a repair. Most OS files aren’t likely to be damaged during an abrupt power outage, but it’s not guaranteed. However, /home is an entirely different situation where loss of important data is highly likely.
Given OpenBSD’s reputation as being a rock-solid OS, and Unix-like OSes in general, the odds of Syncthing (a userland process) directly causing a kernel panic is extremely low. If anything, it hints at hardware issues such as bad storage media, flaky RAM, power issue, overheating, etc.
