Cause and effect may be reversed here, as well. The previous (Syncthing) crash you had essentially says “I tried to read a list of 1507762662 items into a field that’s supposed to never contain more than 64 items; crashing” (probably something like a folder ID, which is limited to 64 characters). Not all fields have such restrictive limits, so reading corrupt data can use rather a lot of memory… I’m sorry, but I just don’t think your system is any good for debugging this.
Indeed, there might be a hardware issue which I’ve not been able to identify (yet). PSU and memory haven’t been swapped out, all other components have been.
Just a thought : Something is corrupting the DB, right? Why do all hosts in the sync agree that everything is up to date and properly synced ? With a corrupted DB on 1 of the hosts, this shouldn’t be possible…
Depends what got corrupted 
well, if corruption doesn’t affect proper synching, then the corruption shouldn’t matter, and ST should ignore it, and not crash.
That’s not the case. If for example the list of block hashes gets corrupted but the version number or modified time doesn’t, syncthing doesn’t have a reason to refer to it so the change goes undetected. And so on.
why is there such a big discrepancy between memory usage as shown in the webif (5.31GB)versus what the system (6.45GB) is showing ? Referring to screenie higher up. On other hosts these figures match up pretty closely, give or take a couple of MB’s
FWIW, I’ve ruled out memory as a possible cause for trouble. Ran http://www.memtest.org/ for 13 hours, without any errors. Next up is the PSU, and for good measure, I am transferring all 4.5TB of data onto other disks and will rebuild the entire DB.
I recently started running into situations where syncthing will produce a panic and crash, then resulting in a continuous loop of panic and crash at startup. After posting in the forums, I was directed to this thread. From what I can tell, it’s just an unavoidable bug in the program and the only solution is to install a version with a different database backend?
Is that a correct assessment, or am I missing something?
You’re probably missing something. What’s the crash log? The point of the alternate database backend is to explore options, but the actual cause of the crashes seem to be related to hardware or driver problems as far as I can tell.
This is the error thrown as it shows up in SyncTrayzor:
[PVWN3] 01:56:34 INFO: syncthing v0.11.16 (go1.4.2 windows-amd64 default) unknown-user@syncthing-builder 2015-07-19 11:34:11 UTC [PVWN3] 01:56:34 INFO: My ID: PVWN3RI [PVWN3] 01:56:34 INFO: Local networks: 192.168.1.114/24 [PVWN3] 01:56:34 INFO: Database block cache capacity 65536 KiB panic: leveldb/table: corruption on data-block (pos=0): checksum mismatch, want=0x2603e07b got=0x2e50a6f [file=180399.ldb] goroutine 1 [running]: github.com/syncthing/syncthing/internal/db.ldbCheckGlobals(0xc08228a160, 0xc0834b9e90, 0xb, 0x10) /go/src/github.com/syncthing/syncthing/internal/db/leveldb.go:1053 +0xde7 github.com/syncthing/syncthing/internal/db.NewFileSet(0xc082003f94, 0xb, 0xc08228a160, 0xc082668350) /go/src/github.com/syncthing/syncthing/internal/db/set.go:55 +0x28c github.com/syncthing/syncthing/internal/model.(*Model).AddFolder(0xc082246b40, 0xc082003f94, 0xb, 0xc0820bb600, 0xe, 0xc0820a0200, 0x2, 0x4, 0x0, 0x258, …) /go/src/github.com/syncthing/syncthing/internal/model/model.go:1172 +0x1a8 main.syncthingMain() /go/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:621 +0x221d main.main() /go/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:383 +0x23da goroutine 5 [syscall]: os/signal.loop() c:/go/src/os/signal/signal_unix.go:21 +0x26 created by os/signal.init·1 c:/go/src/os/signal/signal_unix.go:27 +0x3c goroutine 7 [chan receive]: main.trackCPUUsage() /go/src/github.com/syncthing/syncthing/cmd/syncthing/gui_windows.go:37 +0x496 created by main.init·2 /go/src/github.com/syncthing/syncthing/cmd/syncthing/gui_windows.go:17 +0x2c goroutine 8 [select]: github.com/thejerf/suture.(*Supervisor).Serve(0xc0820de000) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/thejerf/suture/suture.go:411 +0xf6b created by github.com/thejerf/suture.(*Supervisor).ServeBackground /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/thejerf/suture/suture.go:373 +0x39 goroutine 10 [select]: github.com/syncthing/syncthing/internal/events.(*Subscription).Poll(0xc082009960, 0xdf8475800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, …) /go/src/github.com/syncthing/syncthing/internal/events/events.go:186 +0x370 github.com/syncthing/syncthing/internal/events.(*BufferedSubscription).pollingLoop(0xc08200c690) /go/src/github.com/syncthing/syncthing/internal/events/events.go:223 +0x4f created by github.com/syncthing/syncthing/internal/events.NewBufferedSubscription /go/src/github.com/syncthing/syncthing/internal/events/events.go:217 +0x299 goroutine 11 [select]: github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc08209e1c0) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x225 created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x253 goroutine 12 [select]: github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc08228a160) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x561 created by github.com/syndtr/goleveldb/leveldb.openDB /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x7d3 goroutine 13 [select]: github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc08228a160) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x151 created by github.com/syndtr/goleveldb/leveldb.openDB /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7ed goroutine 14 [runnable, locked to thread]: syscall.GetConsoleMode(0x1f4, 0xc08441addc, 0x0, 0x0) c:/go/src/syscall/zsyscall_windows.go:1237 +0x77 os.newFile(0x1f4, 0xc082262140, 0x41, 0xc0820074c0) c:/go/src/os/file_windows.go:52 +0xec os.NewFile(0x1f4, 0xc082262140, 0x41, 0x0) c:/go/src/os/file_windows.go:65 +0x5d os.openFile(0xc082262140, 0x41, 0x0, 0x0, 0xc0834b8070, 0x0, 0x0) c:/go/src/os/file_windows.go:88 +0xca os.OpenFile(0xc082262140, 0x41, 0x0, 0x0, 0x0, 0x0, 0x0) c:/go/src/os/file_windows.go:140 +0x1ca github.com/syndtr/goleveldb/leveldb/storage.(*file).Open(0xc082104780, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:392 +0x17b github.com/syndtr/goleveldb/leveldb.func·031(0xc082a4ff30, 0x0, 0x0) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table.go:345 +0xb2 github.com/syndtr/goleveldb/leveldb/cache.(*Cache).Get(0xc082007cc0, 0x0, 0x2c5e0, 0xc08441b0f8, 0x0) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/cache/cache.go:385 +0x1f4 github.com/syndtr/goleveldb/leveldb.(*tOps).open(0xc082008ee0, 0xc0821b2e60, 0x0, 0x0, 0x0) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table.go:363 +0xe5 github.com/syndtr/goleveldb/leveldb.(*tOps).newIterator(0xc082008ee0, 0xc0821b2e60, 0x0, 0xc083c7a0b0, 0x0, 0x0) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table.go:404 +0x5f github.com/syndtr/goleveldb/leveldb.(*tFilesArrayIndexer).Get(0xc083c7c380, 0x4, 0x0, 0x0) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table.go:265 +0xc0 github.com/syndtr/goleveldb/leveldb/iterator.(*arrayIteratorIndexer).Get(0xc082122690, 0x0, 0x0) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/iterator/array_iter.go:164 +0x82 github.com/syndtr/goleveldb/leveldb/iterator.(*indexedIterator).setData(0xc083d180c0) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:39 +0x70 github.com/syndtr/goleveldb/leveldb/iterator.(*indexedIterator).Next(0xc083d180c0, 0x465f39) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:160 +0x138 github.com/syndtr/goleveldb/leveldb/iterator.(*mergedIterator).Next(0xc0820a0900, 0xc083292100) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:169 +0x321 github.com/syndtr/goleveldb/leveldb.(*tableCompactionBuilder).run(0xc082176aa0, 0xc083c7a0a0, 0x0, 0x0) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:411 +0x48a github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransact(0xc08228a160, 0xab4970, 0xb, 0x2fd4d10, 0xc082176aa0) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:164 +0x27e github.com/syndtr/goleveldb/leveldb.(*DB).tableCompaction(0xc08228a160, 0xc083302000, 0x0) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:552 +0x1083 github.com/syndtr/goleveldb/leveldb.(*DB).tableAutoCompaction(0xc08228a160) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:597 +0x57 github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc08228a160) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:789 +0x3a7 created by github.com/syndtr/goleveldb/leveldb.openDB /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x9d6 goroutine 15 [select]: github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc08228a160) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x28a created by github.com/syndtr/goleveldb/leveldb.openDB /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x9f0 goroutine 16 [select]: github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0xc08228a160) /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x19e created by github.com/syndtr/goleveldb/leveldb.openDB /go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:148 +0xa0a goroutine 19 [select]: github.com/syncthing/syncthing/internal/model.(*ProgressEmitter).Serve(0xc082a33d80) /go/src/github.com/syncthing/syncthing/internal/model/progressemitter.go:52 +0x8f9 created by github.com/syncthing/syncthing/internal/model.NewModel /go/src/github.com/syncthing/syncthing/internal/model/model.go:144 +0xb8c
Decided to rebuild the database again yesterday. Got through about 1/4 of the data before throwing this error: http://pastebin.com/4qxMJ1u0
Restarting syncthing at this point gives me the same panic as before (different position this time)
panic: leveldb/table: corruption on data-block (pos=547373): checksum mismatch, want=0x2b44f29f got=0x4d6e8f13 [file=017644.ldb]
Do give the bolt build a try. But the most likely cause here is that a flaky disk, driver or memory is corrupting the data we’re writing to or reading from the database.
or, syncthing is eating up all memory without releasing it fast enough. 