How do I repair the index database?

markshep · January 31, 2021, 3:46pm

I’m getting server panics with the error “device present in global list but missing as device/fileinfo entry” and there’s a “index-v0.14.0.db.needsrepair” file present.

Please can someone tell me how to repair the index database so that Syncthing doesn’t panic any more?

imsodin · January 31, 2021, 4:04pm

Can you post the log before the panic please. There should be a warning with an error that contains information on where this problem occurs.

The .needsrepair file should cause a db check and repair on startup. So either that doesn’t work or the error occurs during said check&repair.

markshep · January 31, 2021, 5:39pm

Here’s the relevant bit from the logs:

[LETOB] WARNING: Fatal error: pnmfz-36wem Drop(HS4SPRW-3CPGSEF-52LDZAO-EZ3SFE6-E7E2YVT-6E72S3U-GMAVMQQ-4IUU3AF): device 
present in global list but missing as device/fileinfo entry
Panic at 2021-01-31T15:10:31Z
panic: device present in global list but missing as device/fileinfo entry

imsodin · January 31, 2021, 6:37pm

Ok, that means it doesn’t happen during repair at startup. Which means it should get repaired when it starts again after this crash. There’s a line in the log indicating if this happened. Can you please post the log with a few lines before the panic plus the first part of the log of when it starts again after crashing (removing filenames etc., if necessary and present). It’s easier to diagnose and help you than to ask for specific lines one after the other.

markshep · February 1, 2021, 11:00pm

Here’s the last panic up and including to the first callstack:

[start] INFO: syncthing v1.12.1 "Fermium Flea" (go1.15.6 linux-arm) deb@build.syncthing.net 2020-12-06 12:46:27 UTC [noupgrade]
[start] INFO: Database was marked for repair - this may take a while
[LETOB] INFO: My ID: LETOB3N-ZJUZXZT-GLWT22B-CLX6BSH-OGKJ3H4-FA7DOWH-Z7RENBZ-2WO6NAI
[LETOB] INFO: Single thread SHA256 performance is 45 MB/s using minio/sha256-simd (45 MB/s using crypto/sha256).
[LETOB] INFO: Hashing performance is 40.48 MB/s
[LETOB] INFO: Overall send rate is unlimited, receive rate is unlimited
[LETOB] INFO: Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[LETOB] INFO: Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[LETOB] INFO: Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[LETOB] INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027
...
[LETOB] INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
[LETOB] INFO: QUIC listener ([::]:22000) starting
[LETOB] INFO: TCP listener ([::]:22000) starting
[LETOB] INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
[LETOB] INFO: GUI and API listening on [::]:8384
[LETOB] INFO: Access the GUI via the following URL: http://127.0.0.1:8384/
[LETOB] INFO: My name is "rpi-4b"
[LETOB] INFO: Device HS4SPRW-3CPGSEF-52LDZAO-EZ3SFE6-E7E2YVT-6E72S3U-GMAVMQQ-4IUU3AF is "synology" at [dynamic]
[LETOB] INFO: Ready to synchronize "projects" (a4dtg-k6tve) (sendreceive)
[LETOB] INFO: Established secure connection to HS4SPRW-3CPGSEF-52LDZAO-EZ3SFE6-E7E2YVT-6E72S3U-GMAVMQQ-4IUU3AF at 192.168.1.113:22000-192.168.1.111:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
[LETOB] INFO: Device HS4SPRW-3CPGSEF-52LDZAO-EZ3SFE6-E7E2YVT-6E72S3U-GMAVMQQ-4IUU3AF client is "syncthing v1.12.1" named "synology" at 192.168.1.113:22000-192.168.1.111:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256
[LETOB] INFO: Device HS4SPRW-3CPGSEF-52LDZAO-EZ3SFE6-E7E2YVT-6E72S3U-GMAVMQQ-4IUU3AF folder "projects" (a4dtg-k6tve) has mismatching index ID for us (0x40604A8A1DACE522 != 0x7A9FDEDADF5078E7)
[LETOB] INFO: Device HS4SPRW-3CPGSEF-52LDZAO-EZ3SFE6-E7E2YVT-6E72S3U-GMAVMQQ-4IUU3AF folder "projects" (a4dtg-k6tve) has a new index ID (0x7A9FDEDADF5078E7)
[LETOB] INFO: Ready to synchronize "video" (cvz7k-a9gec) (sendreceive)
[LETOB] INFO: Device HS4SPRW-3CPGSEF-52LDZAO-EZ3SFE6-E7E2YVT-6E72S3U-GMAVMQQ-4IUU3AF folder "video" (cvz7k-a9gec) has mismatching index ID for us (0x652C123A53D2B095 != 0x9139F166B81DD88B)
[LETOB] INFO: Device HS4SPRW-3CPGSEF-52LDZAO-EZ3SFE6-E7E2YVT-6E72S3U-GMAVMQQ-4IUU3AF folder "video" (cvz7k-a9gec) has a new index ID (0x9139F166B81DD88B)
[LETOB] WARNING: Fatal error: cvz7k-a9gec Drop(HS4SPRW-3CPGSEF-52LDZAO-EZ3SFE6-E7E2YVT-6E72S3U-GMAVMQQ-4IUU3AF): device present in global list but missing as device/fileinfo entry
Panic at 2021-01-31T20:51:10Z
panic: device present in global list but missing as device/fileinfo entry

goroutine 192 [running]:
github.com/syncthing/syncthing/lib/db.obfuscateAndPanic(0xd216d0, 0x2898c90)
        github.com/syncthing/syncthing/lib/db/lowlevel.go:1171 +0x6c
github.com/syncthing/syncthing/lib/db.fatalError(0xd216d0, 0x2898c90, 0x126195c0, 0x51, 0x362fc00)
        github.com/syncthing/syncthing/lib/db/set.go:500 +0x11c
github.com/syncthing/syncthing/lib/db.(*FileSet).Drop(0x11d7dbc0, 0xc627b93c, 0x22cd13db, 0x401e4b77, 0x297267e2, 0x156bf209, 0x4bfd899f, 0xc82a6066, 0x36a58843)
        github.com/syncthing/syncthing/lib/db/set.go:61 +0x6a4
github.com/syncthing/syncthing/lib/model.(*indexSenderRegistry).addLocked(0x8132e60, 0x28275d0, 0xb, 0x28275e8, 0x5, 0x0, 0x2827600, 0xb, 0x0, 0x1260cb40, ...)
        github.com/syncthing/syncthing/lib/model/indexsender.go:302 +0x8d4
github.com/syncthing/syncthing/lib/model.(*indexSenderRegistry).add(0x8132e60, 0x28275d0, 0xb, 0x28275e8, 0x5, 0x0, 0x2827600, 0xb, 0x0, 0x1260cb40, ...)
        github.com/syncthing/syncthing/lib/model/indexsender.go:241 +0x54
github.com/syncthing/syncthing/lib/model.(*model).ccHandleFolders(0x29c6c40, 0xef3c240, 0x6, 0x8, 0xc627b93c, 0x22cd13db, 0x401e4b77, 0x297267e2, 0x156bf209, 0x4bfd899f, ...)
        github.com/syncthing/syncthing/lib/model/model.go:1319 +0x4d8
github.com/syncthing/syncthing/lib/model.(*model).ClusterConfig(0x29c6c40, 0xc627b93c, 0x22cd13db, 0x401e4b77, 0x297267e2, 0x156bf209, 0x4bfd899f, 0xc82a6066, 0x36a58843, 0xef3c240, ...)
        github.com/syncthing/syncthing/lib/model/model.go:1186 +0x510
github.com/syncthing/syncthing/lib/protocol.(*rawConnection).dispatcherLoop(0x844a630, 0x0, 0x0)
        github.com/syncthing/syncthing/lib/protocol/protocol.go:429 +0x894
github.com/syncthing/syncthing/lib/protocol.(*rawConnection).Start.func2(0x844a630)
        github.com/syncthing/syncthing/lib/protocol/protocol.go:262 +0x1c
created by github.com/syncthing/syncthing/lib/protocol.(*rawConnection).Start
        github.com/syncthing/syncthing/lib/protocol/protocol.go:261 +0x64

That’s followed by another 117 callstacks.

There are six folders in all - in addition to “projects” and “video” mentioned in the log file there’s also “personal”, “photos”, “software” and “sync”. I have two instances of syncthing connected together - “synology” (which is my home NAS device) and “rpi-4b” (which I use as a backup/portable NAS) and is where the problem is.

The latter had been turned off for a few months and upon booting it up again it had about 1.5 TB of data to sync from synology. It had made good progress on that (about 600 GB I think), but then suddenly lost power due to a dodgy connector and presumably corrupted the database in the process.

I have now noticed that back on synology lots of copies of files have started appearing in folders which they don’t belong in. Looking in the Syncthing log on the NAS there are lots of lines like this:

[HS4SP] 22:13:20 INFO: Puller (folder "video" (cvz7k-a9gec), item "(2013-02-04) Thailand/DSC01936_IMG_20130216_092312.jpg"): syncing: no connected device has the required version of this file                                                                                                                                                                                                                                                                    
[HS4SP] 22:13:20 INFO: Puller (folder "video" (cvz7k-a9gec), item "(2012-02-26) India - Varanasi to Banbassa/DSC00099.JPG"): syncing: no connected device has the required version of this file                                                                                                                                                                                                                                                                    
[HS4SP] 22:13:20 INFO: Puller (folder "video" (cvz7k-a9gec), item "(2019-08-15) Brittany/IMG_20190821_114426.jpg"): syncing: no connected device has the required version of this file                                                                                                                                                                                                                                                                             
[HS4SP] 22:13:20 INFO: Puller (folder "video" (cvz7k-a9gec), item "(2016-05-14) Naomi/DSC_0015.JPG"): syncing: no connected device has the required version of this file
[HS4SP] 22:13:20 INFO: Puller (folder "video" (cvz7k-a9gec), item "(2012-02-26) India - Varanasi to Banbassa/AAA_IMG_4410.JPG"): syncing: no connected device has the required version of this file                                                                                                                                                                                                                                                                
[HS4SP] 22:13:20 INFO: Puller (folder "video" (cvz7k-a9gec), item "(2020-05-30) Garden/DSC_0036.JPG"): syncing: no connected device has the required version of this file
[HS4SP] 22:13:20 INFO: Puller (folder "video" (cvz7k-a9gec), item "(2013-02-04) Thailand/DSC02189.JPG"): syncing: no connected device has the required version of this file
[HS4SP] 22:13:20 INFO: Puller (folder "video" (cvz7k-a9gec), item "(2019-08-15) Brittany/IMG_20190818_142510.jpg"): syncing: no connected device has the required version of this file

Those files live in the “photos” folder but somehow it thinks there should also be copied of them in the “video” folder now too and this cross-pollination is happening between most of the folders. Maybe somehow Syncthing on rpi-4b combined the indexes from multiple folders and told Syncthing on synology about them before I stopped it running?

I’ll leave rpi-4b alone for now in case you require any more information from it in order to help diagnose the bug.

As synology seems to now think it should have all sorts of extra files I think I should wipe Synthing’s indexes on there, remove the duplicate files, then rebuild the indexes from scratch.

imsodin · February 4, 2021, 11:43am

Thanks for all the info!

Unfortunately a bunch of info I’d need is not logged. For example the initial db repair clearly does kick in, but doesn’t report whether it fixed anything.

That’s great! You could also create a copy of the database folder (index-v0.14.0.db), and then do whatever to resolve the problem afterwards.

I don’t really see how db corruption should lead to combined/switched folder. I’d tend to be more suspicious about:

Something might have changed in your paths while it was offline, or updates happened on bringing it online that changed structure or something like that.

I am not trying to deflect and saying this is not a Syncthing bug, I just don’t see what it is (yet) and bringing up alternatives.

I will do some changes to this code soon, such that it can handle (self-correct) such problems. For now there’s probably no way around rebuilding indexes to get rid of the corruption/mixing.

system · March 6, 2021, 11:44am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.