Panic on Raspberry Pi

Support
1

One of my Pi’s went offline a few days ago - when I got access to it yesterday, rebooted it, waited for the web gui to respond - it never did. Went into the .config folder and found 6634 panic files…

here is one of them

    [PHDB3] 12:07:18 INFO: syncthing v0.11.10 (go1.4.2 linux-arm default) unknown-user@syncthing-builder 2015-06-21 09:45:54 UTC
    [PHDB3] 12:07:18 INFO: My ID: ********************************************
    [PHDB3] 12:07:18 INFO: Database block cache capacity 8192 KiB
    ...
    Panic at 2015-06-27T12:07:18+01:00
    panic: leveldb: iKey "\x01", len=1: invalid length

    goroutine 1 [running]:
    github.com/syndtr/goleveldb/leveldb.iKey.assert(0x10a44ecc, 0x1, 0x23)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/key.go:109 +0x1d4
    github.com/syndtr/goleveldb/leveldb.iKey.ukey(0x10a44ecc, 0x1, 0x23, 0x0, 0x0, 0x0)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/key.go:114 +0x44
    github.com/syndtr/goleveldb/leveldb.(*iComparer).Compare(0x10aaa458, 0x10a44ecc, 0x1, 0x23, 0x10af4012, 0x9, 0x9, 0xffffffff)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/comparer.go:36 +0x38
    github.com/syndtr/goleveldb/leveldb/table.func·001(0x5, 0x849900)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:68 +0x350
    sort.Search(0x6, 0x10adab9c, 0x0)
	/usr/local/go/src/sort/search.go:66 +0x60
    github.com/syndtr/goleveldb/leveldb/table.(*block).seek(0x10ee6330, 0x76fb5840, 0x10aaa458, 0x0, 0x6, 0x10af4012, 0x9, 0x9, 0x3f0a30, 0x0, ...)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:69 +0x9c
    github.com/syndtr/goleveldb/leveldb/table.(*blockIter).Seek(0x10e76c80, 0x10af4012, 0x9, 0x9, 0x0)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:229 +0xe8
    github.com/syndtr/goleveldb/leveldb/table.(*Reader).newBlockIter(0x113eb290, 0x10ee6330, 0x76fb6cf0, 0x10a0bbc8, 0x10b3ec00, 0x1, 0x0)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:736 +0x1d0
    github.com/syndtr/goleveldb/leveldb/table.(*Reader).NewIterator(0x113eb290, 0x10b3ec00, 0x0, 0x0, 0x0)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:804 +0x2e4
    github.com/syndtr/goleveldb/leveldb.(*tOps).newIterator(0x10a39690, 0x10b49890, 0x10b3ec00, 0x0, 0x0, 0x0)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table.go:408 +0x160
    github.com/syndtr/goleveldb/leveldb.(*version).getIterators(0x10b4b380, 0x10b3ec00, 0x0, 0x113e9400, 0x80, 0x80)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/version.go:233 +0xcc
   github.com/syndtr/goleveldb/leveldb.(*DB).newRawIterator(0x10a3e540, 0x10b3ec00, 0x0, 0x0, 0x0)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_iter.go:40 +0x78
    github.com/syndtr/goleveldb/leveldb.(*DB).newIterator(0x10a3e540, 0xcb786dd, 0x0, 0x10b3eba0, 0x0, 0x10af4011)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_iter.go:69 +0x1a0
    github.com/syndtr/goleveldb/leveldb.(*Snapshot).NewIterator(0x10a2f2f0, 0x10b3eba0, 0x0, 0x0, 0x0)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_snapshot.go:161 +0x2e0
github.com/syncthing/syncthing/internal/db.ldbListFolders(0x10a3e540, 0x0, 0x0, 0x0)
	/go/src/github.com/syncthing/syncthing/internal/db/leveldb.go:930 +0x1f8
github.com/syncthing/syncthing/internal/db.ListFolders(0x10a3e540, 0x0, 0x0, 0x0)
	/go/src/github.com/syncthing/syncthing/internal/db/set.go:219 +0x34
main.syncthingMain()
	/go/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:519 +0x17e0
main.main()
	/go/src/github.com/syncthing/syncthing/cmd/syncthing/main.go:271 +0x1d28

goroutine 5 [syscall]:
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:21 +0x1c
created by os/signal.init·1
	/usr/local/go/src/os/signal/signal_unix.go:27 +0x40

goroutine 7 [chan receive]:
main.trackCPUUsage()
	/go/src/github.com/syncthing/syncthing/cmd/syncthing/gui_unix.go:24 +0x120
created by main.init·2
	/go/src/github.com/syncthing/syncthing/cmd/syncthing/gui_unix.go:17 +0x34

goroutine 8 [select]:
github.com/thejerf/suture.(*Supervisor).Serve(0x10adc180)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/thejerf/suture/suture.go:412 +0xd60
created by github.com/thejerf/suture.(*Supervisor).ServeBackground
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/thejerf/suture/suture.go:373 +0x3c

goroutine 10 [select]:
github.com/syncthing/syncthing/internal/events.(*Subscription).Poll(0x10a38e10, 0xf8475800, 0xd, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/syncthing/syncthing/internal/events/events.go:183 +0x2b8
github.com/syncthing/syncthing/internal/events.(*BufferedSubscription).pollingLoop(0x10a2e990)
	/go/src/github.com/syncthing/syncthing/internal/events/events.go:220 +0x40
created by github.com/syncthing/syncthing/internal/events.NewBufferedSubscription
	/go/src/github.com/syncthing/syncthing/internal/events/events.go:214 +0x2a8

goroutine 11 [select]:
github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0x10a6e420)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:205 +0x1d4
created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:236 +0x23c

goroutine 12 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0x10a3e540)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:69 +0x474
created by github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x794

goroutine 13 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0x10a3e540)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_state.go:82 +0x124
created by github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x7bc

goroutine 14 [runnable]:
syscall.Syscall6(0xb4, 0xd3, 0x10f64000, 0x10b, 0x0, 0x292a03, 0x0, 0x10b, 0x10f64000, 0x0)
	/usr/local/go/src/syscall/asm_linux_arm.s:48 +0x8
syscall.Pread(0xd3, 0x10f64000, 0x10b, 0x2ef, 0x292a03, 0x0, 0x10b12340, 0x0, 0x0)
	/usr/local/go/src/syscall/zsyscall_linux_arm.go:1835 +0xa8
os.(*File).pread(0x10bbf238, 0x10f64000, 0x10b, 0x2ef, 0x292a03, 0x0, 0x60948, 0x0, 0x0)
	/usr/local/go/src/os/file_unix.go:201 +0x6c
os.(*File).ReadAt(0x10bbf238, 0x10f64000, 0x10b, 0x2ef, 0x292a03, 0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/file.go:117 +0xb4
github.com/syndtr/goleveldb/leveldb/table.(*Reader).readRawBlock(0x10f5c090, 0x292a03, 0x0, 0x106, 0x0, 0x292a01, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:561 +0xe8
github.com/syndtr/goleveldb/leveldb/table.(*Reader).readBlock(0x10f5c090, 0x292a03, 0x0, 0x106, 0x0, 0x1, 0x1, 0x0, 0x0)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:599 +0x5c
github.com/syndtr/goleveldb/leveldb/table.(*Reader).readBlockCached(0x10f5c090, 0x292a03, 0x0, 0x106, 0x0, 0x3f0001, 0x10b9b720, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:644 +0x22c
github.com/syndtr/goleveldb/leveldb/table.(*Reader).getIndexBlock(0x10f5c090, 0x0, 0xb70ee08b, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:708 +0x7c
github.com/syndtr/goleveldb/leveldb/table.(*Reader).NewIterator(0x10f5c090, 0x0, 0x10b82090, 0x0, 0x0)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:799 +0x1b4
github.com/syndtr/goleveldb/leveldb.(*tOps).newIterator(0x10a39690, 0x10b1cc90, 0x0, 0x10b82090, 0x0, 0x0)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/table.go:408 +0x160
github.com/syndtr/goleveldb/leveldb.(*compaction).newIterator(0x10bc2000, 0x0, 0x0)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/session_compaction.go:278 +0x268
github.com/syndtr/goleveldb/leveldb.(*tableCompactionBuilder).run(0x10aa4120, 0x10b82078, 0x0, 0x0)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:409 +0x400
github.com/syndtr/goleveldb/leveldb.(*DB).compactionTransact(0x10a3e540, 0x548d50, 0xb, 0x76fb6d88, 0x10aa4120)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:164 +0x21c
github.com/syndtr/goleveldb/leveldb.(*DB).tableCompaction(0x10a3e540, 0x10bc2000, 0x0)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:532 +0xfbc
github.com/syndtr/goleveldb/leveldb.(*DB).tableAutoCompaction(0x10a3e540)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:597 +0x4c
github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0x10a3e540)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:789 +0x2d8
created by github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:154 +0x9a4

goroutine 15 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0x10a3e540)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:715 +0x248
created by github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:154 +0x9cc

goroutine 16 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).jWriter(0x10a3e540)
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db_write.go:37 +0x17c
created by github.com/syndtr/goleveldb/leveldb.openDB
	/go/src/github.com/syncthing/syncthing/Godeps/_workspace/src/github.com/syndtr/goleveldb/leveldb/db.go:154 +0x9f4

Any advice? I have a feeling I will have to delete the index files to make it work again?

2

Yeah try that, but this is the first time I’ve seen this, so perhaps worth an issue on github.

3

https://github.com/syncthing/syncthing/issues/2008 - bug reported here

4

Same pi, 3004 panic files, it dropped out of sync 5 days ago - I had deleted the index to try and let it rebuild. not sure if it ever managed to

5

Not all panic files are the same, the first one is very large, best way of uploading it?

6

Pastebin?

7

This was the first panic … I can give you a sample of some of the others

http://pastebin.com/WkhDLAD4

Most recent:

http://pastebin.com/tXSt9yZW

8

The other ones are just corruption. You can search the forums/github tracker on how to resolve that.

9

What I found was just to delete the indexes and start again? But this is how i got back into the situation? Is the first one also just corruption?

10

The first one is strange, I have never seen it before, hence I cannot say why it happens. If you are constantly getting panics about corrupt indexes, it might be to hardware/software issues outside of syncthing, as explained in the threads.