Folders are Unknown and not refresh

There are 4 of 6 folders status as Unknown on one of 4 syncthing nodes.

I deployed a cluster of syncthing nodes on 4 ARM boards, and setup folders to sync. There are 6 folders to sync on each node. The cluster ran well for days and managed TBs of data. But now there are 4 folders as Unknown on node0. I tried to rescan but not help.

syncthing ver: 1.28.0-1 OS: Linux 6.11.3-1-armv7-ARCH

The ID of Unknown folders are: civnf-fgzct, hvefp-nk526, gd72z-5geto, wke6e-5meu6 I grep the log files, and got errors for some folders as:

[cicada@node0 syncthing]$ grep civnf-fgzct  panic-20241020-165*
panic-20241020-165004.reported.log:[R6W7O] INFO: Ready to synchronize "0-1" (civnf-fgzct) (sendonly)
panic-20241020-165004.reported.log:[R6W7O] WARNING: Fatal error: civnf-fgzct Get(bak/jingju/guchenghui-1.wmv): filling Blocks: leveldb/table: corruption on data-block (pos=573531): checksum mismatch, want=0x91b1ab1 got=0x311db657 [file=006557.ldb]
panic-20241020-165035.reported.log:[R6W7O] INFO: Ready to synchronize "0-1" (civnf-fgzct) (sendonly)
panic-20241020-165035.reported.log:[R6W7O] WARNING: Fatal error: civnf-fgzct Get(bak/jingju/guchenghui-1.wmv): filling Blocks: leveldb/table: corruption on data-block (pos=573531): checksum mismatch, want=0x91b1ab1 got=0x311db657 [file=006557.ldb]
panic-20241020-165106.reported.log:[R6W7O] INFO: Ready to synchronize "0-1" (civnf-fgzct) (sendonly)
panic-20241020-165106.reported.log:[R6W7O] WARNING: Fatal error: civnf-fgzct Get(bak/jingju/guchenghui-1.wmv): filling Blocks: leveldb/table: corruption on data-block (pos=573531): checksum mismatch, want=0x91b1ab1 got=0x311db657 [file=006557.ldb]
panic-20241020-165138.reported.log:[R6W7O] INFO: Ready to synchronize "0-1" (civnf-fgzct) (sendonly)
panic-20241020-165138.reported.log:[R6W7O] WARNING: Fatal error: civnf-fgzct Get(bak/jingju/guchenghui-1.wmv): filling Blocks: leveldb/table: corruption on data-block (pos=573531): checksum mismatch, want=0x91b1ab1 got=0x311db657 [file=006557.ldb]
panic-20241020-165209.reported.log:[R6W7O] INFO: Ready to synchronize "0-1" (civnf-fgzct) (sendonly)
panic-20241020-165209.reported.log:[R6W7O] WARNING: Fatal error: civnf-fgzct Get(bak/jingju/guchenghui-1.wmv): filling Blocks: leveldb/table: corruption on data-block (pos=573531): checksum mismatch, want=0x91b1ab1 got=0x311db657 [file=006557.ldb]

but also some folders looks well, as:

[cicada@node0 syncthing]$ grep wke6e-5meu6  panic-20241020-165*
panic-20241020-165004.reported.log:[R6W7O] INFO: Ready to synchronize "2-0" (wke6e-5meu6) (receiveonly)
panic-20241020-165004.reported.log:[R6W7O] INFO: Completed initial scan of receiveonly folder "2-0" (wke6e-5meu6)
panic-20241020-165035.reported.log:[R6W7O] INFO: Ready to synchronize "2-0" (wke6e-5meu6) (receiveonly)
panic-20241020-165035.reported.log:[R6W7O] INFO: Completed initial scan of receiveonly folder "2-0" (wke6e-5meu6)
panic-20241020-165106.reported.log:[R6W7O] INFO: Ready to synchronize "2-0" (wke6e-5meu6) (receiveonly)
panic-20241020-165106.reported.log:[R6W7O] INFO: Completed initial scan of receiveonly folder "2-0" (wke6e-5meu6)
panic-20241020-165138.reported.log:[R6W7O] INFO: Ready to synchronize "2-0" (wke6e-5meu6) (receiveonly)
panic-20241020-165138.reported.log:[R6W7O] INFO: Completed initial scan of receiveonly folder "2-0" (wke6e-5meu6)
panic-20241020-165209.reported.log:[R6W7O] INFO: Ready to synchronize "2-0" (wke6e-5meu6) (receiveonly)
panic-20241020-165209.reported.log:[R6W7O] INFO: Completed initial scan of receiveonly folder "2-0" (wke6e-5meu6)

Could anyone help? If you need more logs or settings, please let me know.

Thanks in advanced.

You have panic logs, which means Syncthing is crashing. Uploading one of those may shed some light.

Your database has become corrupted. Possible causes are abrupt shutdowns, hardware errors or filesystem corruption. There’s usually no repair for a dead database - wiping the database but keeping config and keys is usually the way to go.

3 Likes

Yes it has been shutdown abnormal due to power failure. I’m not sure if this were the reason. I attached one of the logs panic-20241020-165209.reported.log (92.8 KB)

Yeah, you need to reset your database. Either start syncthing once with syncthing --reset-database, or manually delete the index-v0.14.0.db folder (within syncthing’s data directory) while syncthing is stopped. In both cases, start syncthing normally afterwards. Syncthing will build a new database, but keeping configuration intact. Rebuilding the database may cause an unusual amount of conflict files while the device gets up to speed with the others.

1 Like

Cool! The reset does help. It works at least the folder is well now and keep syncing. But all the synced files need to re-sync, which I believe should be because of the reset of DB.

When the Syncthing database is deleted on a Device, every file needs to be re-hashed, which is resource-intensive on that Device. After that, if the Device has all the files, you’ll see “sync” traffic which is really the Device learning what the Global Model is, and discovering that it has what it needs.

So while it looks like a resync, it’s not very network-intensive.