Updated one device offline to 0.14.2, took 24 hours to reindex. Now another device is online and building his index. Since then syncthing crashes and reboots. I think this is related to v0.14.2 crashes a few minutes after start #3448
Panic : Panic at 2016-07-27T16:30:20+02:00 panic: writing to temporary database: write /tmp/syncthing-db.245441880/000100.ldb: no space left on device
goroutine 170 [running]:
panic(0x5e5490, 0x12113740)
/usr/local/go/src/runtime/panic.go:481 +0x330
github.com/syncthing/syncthing/lib/model.(*onDiskIndexSorter).Append(0x111dd3a0, 0x112ccf80, 0x32, 0x0, 0x1303f17, 0x0, 0x1ff, 0x55be0ee4, 0x0, 0x10000, ...)
/home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/model/sorter.go:166 +0x3b8
github.com/syncthing/syncthing/lib/model.(*autoSwitchingIndexSorter).Append(0x12d16188, 0x112ccf80, 0x32, 0x0, 0x1303f17, 0x0, 0x1ff, 0x55be0ee4, 0x0, 0x10000, ...)
/home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/model/sorter.go:65 +0x21c
github.com/syncthing/syncthing/lib/model.sendIndexTo.func1(0x3119a9d8, 0x11b80690, 0xc084b8)
/home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/model/model.go:1273 +0x1ec
github.com/syncthing/syncthing/lib/db.nativeFileIterator.func1(0x3119a9d8, 0x11b80640, 0x2000)
/home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/db/set.go:323 +0x1fc
github.com/syncthing/syncthing/lib/db.(*Instance).withHave(0x1121dce0, 0x10fa7d70, 0xb, 0xb, 0x10fa7de4, 0x20, 0x20, 0x0, 0x0, 0x0, ...)
/home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/db/leveldb_dbinstance.go:260 +0x54c
github.com/syncthing/syncthing/lib/db.(*FileSet).WithHave(0x1115eae0, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff, 0x10e0b1a0)
/home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/db/set.go:194 +0x22c
github.com/syncthing/syncthing/lib/model.sendIndexTo(0x0, 0x0, 0x30f79100, 0x12ce44c0, 0x12f96784, 0xb, 0x1115eae0, 0x12069d70, 0x0, 0x0, ...)
/home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/model/model.go:1275 +0x3bc
github.com/syncthing/syncthing/lib/model.sendIndexes(0x30f79100, 0x12ce44c0, 0x12f96784, 0xb, 0x1115eae0, 0x12069d70, 0x0, 0x0)
/home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/model/model.go:1214 +0x49c
created by github.com/syncthing/syncthing/lib/model.(*Model).ClusterConfig
/home/buildslave/workspace/syncthing-release/src/github.com/syncthing/syncthing/lib/model/model.go:733 +0x1244
Corresponding (redacted) log :
sh-4.3$ /usr/local/syncthing/bin/syncthing -logflags="11" -verbose -audit --home /usr/local/syncthing/var/
[monitor] 2016/07/27 16:27:42 cmd/syncthing/monitor.go:94: INFO: Starting syncthing
[start] 2016/07/27 16:27:42 cmd/syncthing/main.go:914: INFO: Audit log in /usr/local/syncthing/var//audit-20160727-162742.log
[SRQKF] 2016/07/27 16:27:42 cmd/syncthing/main.go:569: INFO: syncthing v0.14.2 "Dysprosium Dragonfly" (go1.6.3 linux-arm) unknown-user@lx-buildslave.syncthing.net 2016-07-26 11:55:25 UTC
[SRQKF] 2016/07/27 16:27:42 cmd/syncthing/main.go:570: INFO: My ID: MY_ID
[SRQKF] 2016/07/27 16:27:43 cmd/syncthing/main.go:851: INFO: Single thread hash performance is ~7.4 MB/s
[SRQKF] 2016/07/27 16:27:43 cmd/syncthing/verboseservice.go:48: VERBOSE: Starting up (/usr/local/syncthing/var/)
[SRQKF] 2016/07/27 16:29:28 lib/model/model.go:237: INFO: Ready to synchronize SharedFolder (readwrite)
[SRQKF] 2016/07/27 16:29:28 lib/model/model.go:237: INFO: Ready to synchronize SharedFolderID (readwrite)
[SRQKF] 2016/07/27 16:29:28 cmd/syncthing/main.go:705: INFO: Using discovery server https://discovery-v4-1.syncthing.net/v2/?id=SR7AARM-TCBUZ5O-VFAXY4D-CECGSDE-3Q6IZ4G-XG7AH75-OBIXJQV-QJ6NLQA
[SRQKF] 2016/07/27 16:29:28 cmd/syncthing/main.go:705: INFO: Using discovery server https://discovery-v4-2.syncthing.net/v2/?id=DVU36WY-H3LVZHW-E6LLFRE-YAFN5EL-HILWRYP-OC2M47J-Z4PE62Y-ADIBDQC
[SRQKF] 2016/07/27 16:29:28 cmd/syncthing/main.go:705: INFO: Using discovery server https://discovery-v4-3.syncthing.net/v2/?id=VK6HNJ3-VVMM66S-HRVWSCR-IXEHL2H-U4AQ4MW-UCPQBWX-J2L2UBK-NVZRDQZ
[SRQKF] 2016/07/27 16:29:28 cmd/syncthing/main.go:705: INFO: Using discovery server https://discovery-v6-1.syncthing.net/v2/?id=SR7AARM-TCBUZ5O-VFAXY4D-CECGSDE-3Q6IZ4G-XG7AH75-OBIXJQV-QJ6NLQA
[SRQKF] 2016/07/27 16:29:28 cmd/syncthing/main.go:705: INFO: Using discovery server https://discovery-v6-2.syncthing.net/v2/?id=DVU36WY-H3LVZHW-E6LLFRE-YAFN5EL-HILWRYP-OC2M47J-Z4PE62Y-ADIBDQC
[SRQKF] 2016/07/27 16:29:28 cmd/syncthing/main.go:705: INFO: Using discovery server https://discovery-v6-3.syncthing.net/v2/?id=VK6HNJ3-VVMM66S-HRVWSCR-IXEHL2H-U4AQ4MW-UCPQBWX-J2L2UBK-NVZRDQZ
[SRQKF] 2016/07/27 16:29:28 lib/connections/tcp_listen.go:69: INFO: TCP listener ([::]:22000) starting
[SRQKF] 2016/07/27 16:29:28 cmd/syncthing/verboseservice.go:48: VERBOSE: Folder "SharedFolderID" is now scanning
[SRQKF] 2016/07/27 16:29:28 cmd/syncthing/verboseservice.go:48: VERBOSE: Folder "SharedFolder" is now scanning
[SRQKF] 2016/07/27 16:29:28 lib/model/rwfolder.go:299: INFO: Completed initial scan (rw) of folder SharedFolderID
[SRQKF] 2016/07/27 16:29:28 cmd/syncthing/verboseservice.go:48: VERBOSE: Folder "SharedFolderID" is now idle
[SRQKF] 2016/07/27 16:29:29 cmd/syncthing/main.go:750: INFO: Device OTHER_DEVICE_ID_1 is "OTHER_DEVICE_1" at [dynamic]
[SRQKF] 2016/07/27 16:29:29 cmd/syncthing/main.go:750: INFO: Device OTHER_DEVICE_ID_2 is "OTHER_DEVICE_2" at [dynamic]
[SRQKF] 2016/07/27 16:29:29 cmd/syncthing/main.go:750: INFO: Device OTHER_DEVICE_ID_3 is "OTHER_DEVICE_3" at [dynamic]
[SRQKF] 2016/07/27 16:29:29 cmd/syncthing/main.go:750: INFO: Device MY_ID is "MY_DEVICE" at [dynamic]
[SRQKF] 2016/07/27 16:29:29 cmd/syncthing/main.go:750: INFO: Device OTHER_DEVICE_ID_4 is "OTHER_DEVICE_4" at [dynamic]
[SRQKF] 2016/07/27 16:29:29 cmd/syncthing/usage_report.go:264: INFO: Starting usage reporting
[SRQKF] 2016/07/27 16:29:29 cmd/syncthing/verboseservice.go:48: VERBOSE: Startup complete
[SRQKF] 2016/07/27 16:29:29 cmd/syncthing/gui.go:322: INFO: GUI and API listening on [::]:7070
[SRQKF] 2016/07/27 16:29:29 cmd/syncthing/gui.go:323: INFO: Access the GUI via the following URL: http://127.0.0.1:7070/
[SRQKF] 2016/07/27 16:29:30 lib/discover/local.go:159: WARNING: Incompatible (v0.13) local discovery packet from 192.168.1.101:34129 - upgrade that device to connect
[SRQKF] 2016/07/27 16:29:30 lib/discover/local.go:159: WARNING: Incompatible (v0.13) local discovery packet from [SOME_ADDRESS]:48207 - upgrade that device to connect
[SRQKF] 2016/07/27 16:29:31 lib/connections/service.go:619: WARNING: Connecting to "OTHER_DEVICE_3" (syncthing v0.13.10, OTHER_DEVICE_ID_3): the remote device speaks an older version of the protocol (v0.13) not compatible with this version
[SRQKF] 2016/07/27 16:29:33 lib/connections/service.go:252: INFO: Established secure connection to OTHER_DEVICE_ID_1 at 192.168.1.100:22000-IP_DEVICE_1:50281 (TCP (Server))
[SRQKF] 2016/07/27 16:29:33 lib/model/model.go:1105: INFO: Device OTHER_DEVICE_ID_1 client is "syncthing v0.14.2" named "OTHER_DEVICE_NAME_1"
[SRQKF] 2016/07/27 16:29:33 cmd/syncthing/verboseservice.go:48: VERBOSE: Connected to device OTHER_DEVICE_ID_1 at IP_DEVICE_1:50281 (type TCP (Server))
[SRQKF] 2016/07/27 16:29:34 lib/model/model.go:697: INFO: Device OTHER_DEVICE_ID_1 folder "SharedFolder" is delta index compatible (mlv=0)
[SRQKF] 2016/07/27 16:29:36 lib/connections/service.go:619: WARNING: Connecting to "OTHER_DEVICE_2" (syncthing v0.13.10, OTHER_DEVICE_ID_2): the remote device speaks an older version of the protocol (v0.13) not compatible with this version
[SRQKF] 2016/07/27 16:29:37 lib/connections/service.go:205: INFO: Connected to already connected device (OTHER_DEVICE_ID_1)
[SRQKF] 2016/07/27 16:29:38 cmd/syncthing/verboseservice.go:48: VERBOSE: Device OTHER_DEVICE_ID_1 sent an index update for "SharedFolder" with 24 items
[SRQKF] 2016/07/27 16:29:38 cmd/syncthing/verboseservice.go:48: VERBOSE: Device OTHER_DEVICE_ID_1 sent an index update for "SharedFolder" with 36 items
[SRQKF] 2016/07/27 16:29:39 cmd/syncthing/verboseservice.go:48: VERBOSE: Device OTHER_DEVICE_ID_1 sent an index update for "SharedFolder" with 69 items
[SRQKF] 2016/07/27 16:29:40 cmd/syncthing/verboseservice.go:48: VERBOSE: Device OTHER_DEVICE_ID_1 sent an index update for "SharedFolder" with 31 items
[SRQKF] 2016/07/27 16:29:41 cmd/syncthing/verboseservice.go:48: VERBOSE: Device OTHER_DEVICE_ID_1 sent an index update for "SharedFolder" with 31 items
[SRQKF] 2016/07/27 16:29:41 cmd/syncthing/verboseservice.go:48: VERBOSE: Device OTHER_DEVICE_ID_1 sent an index update for "SharedFolder" with 12 items
[SRQKF] 2016/07/27 16:29:42 lib/relay/client/static.go:92: INFO: Joined relay relay://138.201.156.90:22067
[SRQKF] 2016/07/27 16:29:48 cmd/syncthing/verboseservice.go:48: VERBOSE: Listen address dynamic+https://relays.syncthing.net/endpoint resolution has changed: lan addresses: [relay://138.201.156.90:22067/?id=XYWRY6J-I4PVS76-OYBJEK5-MMWAFOQ-4DFHD45-6WVDXVN-N63DPXD-CAZ56AD&pingInterval=1m0s&networkTimeout=2m0s&sessionLimitBps=0&globalLimitBps=300000&statusAddr=:22070&providedBy=aucubin] wan addresses: [relay://138.201.156.90:22067/?id=XYWRY6J-I4PVS76-OYBJEK5-MMWAFOQ-4DFHD45-6WVDXVN-N63DPXD-CAZ56AD&pingInterval=1m0s&networkTimeout=2m0s&sessionLimitBps=0&globalLimitBps=300000&statusAddr=:22070&providedBy=aucubin]
[SRQKF] 2016/07/27 16:29:50 cmd/syncthing/verboseservice.go:48: VERBOSE: Device OTHER_DEVICE_ID_1 sent an index update for "SharedFolder" with 18 items
[SRQKF] 2016/07/27 16:29:51 lib/nat/structs.go:32: INFO: New NAT port mapping: external TCP address MY_IP:60258 to local address 0.0.0.0:22000.
[SRQKF] 2016/07/27 16:29:51 lib/nat/service.go:61: INFO: Detected 2 NAT devices
[SRQKF] 2016/07/27 16:29:51 cmd/syncthing/verboseservice.go:48: VERBOSE: Listen address tcp://0.0.0.0:22000 resolution has changed: lan addresses: [tcp://0.0.0.0:22000] wan addresses: [tcp://0.0.0.0:22000 tcp://0.0.0.0:60258 tcp://0.0.0.0:60258]
[SRQKF] 2016/07/27 16:30:15 cmd/syncthing/verboseservice.go:48: VERBOSE: Device OTHER_DEVICE_ID_1 sent an index update for "SharedFolder" with 13 items
panic: writing to temporary database: write /tmp/syncthing-db.245441880/000100.ldb: no space left on device
[monitor] 2016/07/27 16:30:20 cmd/syncthing/monitor.go:188: WARNING: Panic detected, writing to "/usr/local/syncthing/var//panic-20160727-163020.log"
[monitor] 2016/07/27 16:30:20 cmd/syncthing/monitor.go:189: WARNING: Please check for existing issues with similar panic message at https://github.com/syncthing/syncthing/issues/
[monitor] 2016/07/27 16:30:20 cmd/syncthing/monitor.go:190: WARNING: If no issue with similar panic message exists, please create a new issue with the panic log attached
[monitor] 2016/07/27 16:30:20 cmd/syncthing/monitor.go:163: INFO: Syncthing exited: exit status 2
Can provide more context, logs or traces, here or in issue.