Syncthing won't start after a power outage.

Support
1

Had a power outage last night and now syncthing won’t start. It keeps exiting with error 1. When inspecting the systemd logs I found:

FATAL: Error opening database: open //.config/syncthing/index-v0.14.0.db/LOCK: read-only file system (is another instance of Syncthing running?)

As syncthing went down in an uncontrolled fashion, I went ahead and removed the LOCK file and attempted to restart, but it is still exiting with error code 1. I added -verbose to the start parameters, but that hasn’t yielded anything useful.

Here is its current start logging:

INFO: syncthing v1.0.0 “Erbium Earthworm” (go1.11.4 linux-amd64) deb@build.syncthing.net 2018-12-26 16:23:05 UTC [noupgrade]

INFO: My ID: <snipped, I don’t know if its safe to post the id>

INFO: Single thread SHA256 performance is 417 MB/s using crypto/sha256 (409 MB/s using minio/sha256-simd).

VERBOSE: Starting up (//.config/syncthing)

systemd[1]: syncthing.service: Main process exited, code=exited, status=1/FAILURE

I’ve examined the drive and it appears to be okay. I checked the logs for any fsck message but the drive passed on the reboot. I’ve manually run fsck and no corrections were noted, the drive is clean.

What can I do to get more information to debug a startup failure?

Thanks!

2

Is the directory in question actually writable? What does it fail with after you removed the lock file?

3

What I posted above is what its doing after the removal of the LOCK file. It no longer complains about the FATAL error on the LOCK file, but it continues to exit with error code 1. The db directory is readable and writable to the user that is running syncthing and the permissions on the .config/syncthing directory and files are all still owned and read/write to the user running syncthing.

4

New wrinkle.

If I start syncthing by hand with the arguments from the systemd service file and as the syncthing user, it operates normally. When I start it via the systemd service file it continues to exit with error code 1.

This is really bizarre to me. This was working fine yesterday.

5

You’re running with the config directory at /. That’s an unusual setup, and also implies you’re running as root which we recommend against. Since the last release or two we have hardening options added to the systemd unit which prevent writes like that. You’ll need to take those out or have your stuff in a more usual location.

6

Sorry, the // is where I snipped out the path for posting purposes. I for got to put a note in there.

I found that commenting out the Hardening line ‘ProtectSystem=full’ from the systemd file allows it to start successfully now. I just don’t understand what changed. This power outage is the first time the system has been rebooted in over 60 days. Is it possible an ubuntu update changed how ProtectSystem works and on reboot is now causing problems with syncthing?

7

Old, buggy, systemds exist which make /home readonly instead of the expected parts of the system. If you hide the details of your setup from us it’s hard to guess. But it sounds more likely that your power outage resulted in the Syncthing upgrade, or something like that.

8

Indeed my systemd is older now. This is running on ubuntu 16.04 LTS. I could have sworn I had already upgraded this server to v1.0.0 but maybe not.

9

Turns out the outage is indeed the first time v1.0.0 has been attempted to start. The previous version was running yesterday. I may have done the upgrade, but I never restarted the service.

Its been a really long time since I’ve looked at the syncthing@ systemd service file. Are the hardening rules new? Is the ‘ProtectSystem’ rule new or has it been changed?

10

@grheard

Below link points to the relevant feature request:

And this link is to show the added options to fulfil the request:

11

Thanks for the link. Now it all makes sense.

From the systemd man page:

ProtectSystem: Takes a boolean argument or “full”. If true, mounts the /usr /boot directories read-only for processes invoked by this unit.

In my setup, the syncthing user’s home directory is on a 4TB drive mounted under /usr/local/.

An error message pertaining to the inability to write would have been nice before syncthing exited.

Thanks for everyone’s help.

12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.