Syncthing cannot open database (no lock, no other instance)

Syncthing is giving me the message:

[CNOWE] 2017/11/08 15:15:10 main.go:647: FATAL: Cannot open database: interrupted system call - Is another copy of Syncthing already running?

I’ve checked the common causes/fixes that I am aware of:

  1. No other copy is running.
  2. The disk is not full.
  3. I’ve tried removing the database altogether and having it rebuild from scratch.

I don’t know how to approach debugging beyond this. What are the next steps?

Corrupt binary? Some exotic operating system?

Yes. Details please. Syncthing version, operating system, environment (physical/vim/exotic paravirtualized domain stuff/security frameworks/…), file system, normal user or root, etc?

I’ve never seen this before.

Sorry. Had those in the post, but then decided to rewrite it, and must have left them out.

  • Syncthing v0.14.6
  • CentOS 6.4
  • Physical machine (6 core Xeon, 256 GB ram)
  • The config/database are on an NFS filesystem.
  • Standard, non-root user

I have at least 10 other machines with nearly identical configurations working just fine.

Right now, this is all that makes it into the log, after running for 10 minutes or so:

[monitor] 2017/11/09 07:16:58 monitor.go:94: INFO: Starting syncthing
[CNOWE] 2017/11/09 07:16:58 main.go:568: INFO: syncthing v0.14.6 "Dysprosium Dragonfly" (go1.7 linux-amd64) jenkins@build.syncthing.net 2016-09-04 21:06:30 UTC
[CNOWE] 2017/11/09 07:16:58 main.go:569: INFO: My ID: CNOWE2X-525PZCF-ZBPY7KY-VRUN5KD-JCAINES-OYV2NBI-AO5BEK2-5HMAKAF
[CNOWE] 2017/11/09 07:16:58 main.go:856: INFO: Single thread hash performance is ~171 MB/s
[CNOWE] 2017/11/09 07:16:58 verboseservice.go:48: VERBOSE: Starting up (/home/caduser/bindist/design)

Switching to the latest version (0.14.40) doesn’t help. The version/startup stanza appears twice, which suggests there really are multiple instances trying to write to the log, but I don’t know how that could be.

Here’s the log:

[monitor] 2017/11/09 07:42:26 monitor.go:95: INFO: Starting syncthing
[CNOWE] 2017/11/09 07:42:27 main.go:649: INFO: syncthing v0.14.40 "Dysprosium Dragonfly" (go1.9.2 linux-amd64) teamcity@build.syncthing.net 2017-10-28 19:15:32 UTC
[CNOWE] 2017/11/09 07:42:27 main.go:650: INFO: My ID: CNOWE2X-525PZCF-ZBPY7KY-VRUN5KD-JCAINES-OYV2NBI-AO5BEK2-5HMAKAF
[CNOWE] 2017/11/09 07:42:27 sha256.go:92: INFO: Single thread SHA256 performance is 248 MB/s using minio/sha256-simd (166 MB/s using crypto/sha256).
[CNOWE] 2017/11/09 07:42:27 verboseservice.go:48: VERBOSE: Starting up (/home/caduser/bindist/design)
[CNOWE] 2017/11/09 07:42:28 config.go:415: INFO: Failed to upgrade folder marker: remove /proj/gpfs/spsn_ip/.stfolder: permission denied
[CNOWE] 2017/11/09 07:42:28 main.go:1003: INFO: Archiving a copy of old config file format at: /home/caduser/bindist/design/config.xml.v16
[CNOWE] 2017/11/09 07:42:28 verboseservice.go:48: VERBOSE: Configuration was saved
[CNOWE] 2017/11/09 07:42:28 main.go:709: INFO: Hashing performance with weak hash is 202.72 MB/s
[CNOWE] 2017/11/09 07:42:29 main.go:711: INFO: Hashing performance without weak hash is 234.00 MB/s
[CNOWE] 2017/11/09 07:42:29 main.go:717: INFO: Weak hash enabled, as it has an acceptable performance impact.
[CNOWE] 2017/11/09 07:42:33 main.go:750: FATAL: Cannot open database: interrupted system call - Is another copy of Syncthing already running?
[monitor] 2017/11/09 07:42:33 monitor.go:159: INFO: Syncthing exited: exit status 1
[monitor] 2017/11/09 07:42:34 monitor.go:95: INFO: Starting syncthing
[CNOWE] 2017/11/09 07:42:34 main.go:649: INFO: syncthing v0.14.40 "Dysprosium Dragonfly" (go1.9.2 linux-amd64) teamcity@build.syncthing.net 2017-10-28 19:15:32 UTC
[CNOWE] 2017/11/09 07:42:34 main.go:650: INFO: My ID: CNOWE2X-525PZCF-ZBPY7KY-VRUN5KD-JCAINES-OYV2NBI-AO5BEK2-5HMAKAF
[CNOWE] 2017/11/09 07:42:35 sha256.go:92: INFO: Single thread SHA256 performance is 253 MB/s using minio/sha256-simd (171 MB/s using crypto/sha256).
[CNOWE] 2017/11/09 07:42:35 verboseservice.go:48: VERBOSE: Starting up (/home/caduser/bindist/design)
[CNOWE] 2017/11/09 07:42:36 main.go:709: INFO: Hashing performance with weak hash is 214.05 MB/s
[CNOWE] 2017/11/09 07:42:37 main.go:711: INFO: Hashing performance without weak hash is 246.86 MB/s
[CNOWE] 2017/11/09 07:42:37 main.go:717: INFO: Weak hash enabled, as it has an acceptable performance impact.

Pretty sure you can’t do this, sorry. The database gets mmapped and I don’t think that works over NFS. Or the locking doesn’t. Or the flushing doesn’t. Or something - but I’m pretty sure it’s the problem.

Syncthing, by default, uses a monitor process that manages restarts etc of the real Syncthing process, which is why you see the startup messages multiple times.

Huh. I moved it over to /tmp just to try, and it does appear to be working now.

I’m kind of surprised, because I’ve had syncthing working well with configs on NFS filesystems for about a year and half, and this is the first time this has happened.

I’ll have to see if we changed anything about our NFS configs.

We’ve upgraded the database library which potentially broke this, but to be honest, you shouldn’t be doing this anyway.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.