I have a folder I’m syncing. The total size of the data is 149 GiB (output of du -sh) and it has a count of 12376 files and folders (output of find . | wc -l). At the remote end that is receiving the folder, Syncthing is saying for this folder:
Syncing 36%, 471 GB
Global State 52192 ... 736 GB
Local State 30171 ... 425 GB
Out of date items 38761 ... 469 GB
These numbers don’t seem to relate to the folder. What do they mean?
If I click on the “out of date items” link, all the pages at the end of the list are empty.
(The sending Syncthing shows the correct size and number of items.)
Just after I posted this, the sync stopped. The receiving Syncthing now just says the folder is “Out of sync” in red. It offers no remedy for that. It shows:
Global State 52,192 8,412 ~736 GiB
Local State 31,340 5,050 ~442 GiB
Out of Sync Items 36,321 items, ~441 GiB
If you click on the “Out of Sync Items” link you get a pop up window offering 3633 pages of … well nothing. There are no files shows in any of the pages.
By the way I’ve tried removing and re-adding the folder at the destination, but it just came back the same. I also tried adding it and pointing it a fresh empty directory, so it could re-sync from scratch. Same outcome.
Is my database corrupt? How’d that happen? How do I fix it?
Just a bit more info. The receiving Syncthing shows the sending Syncthing with
Out of Sync items 48,428 items, ~588 GiB
There were never 588 GiB of stuff on the sending end. And if you click on the “48428 items” link you get an “Out of Sync Items” window with nothing in it.
Nevertheless, if possible, please post full screenshots of the Syncthing Web GUI from both sides. Make sure to include the Syncthing version, the operating systems involved, some examples of the “out of sync items”, etc. Basically, the more detailed and specific the information is, the better.
@tomasz86 Thanks for confirming it as a bug and not just me going nuts. I will see about making some screenshots.
About the sending Syncthing says: “v1.22.1, macOS (64-bit Intel/AMD)
“Fermium Flea” Build 2022-11-02 (stnoupgrade)”. It was installed using Homebrew today on to macOS 10.14.6.
About the receiving Syncthing says “v1.22.2, Linux (64-bit Intel/AMD)“Fermium Flea” Build 2022-11-28 (noupgrade)”. It was installed from apt.syncthing.net yesterday on to Ubuntu 20.
I can’t show you any “out of sync items” because Syncthing does not show me any. I don’t believe they exist.
If I run rsync -n -rvis from the source to destination, the only differences I see are for files with UTF-8 normalisation differences (Japanese file names etc.) At first glance there is nothing missing.
I think the destination has some incorrect bookkeeping.
While I’m interested in helping to debug this, I’d also like to know how to fix it locally and move on with other work. Is it possible to run checks on the database integrity? Can I rebuild the database without a lot of manual work in the GUI? It’s important that in future I can recover from bugs without needing to rebuild the entire Syncthing setup in my cluster.
@gadget The source has some other folders but they are paused. The destination has several other folders shared with other devices. The buggy folder is the only folder shared between the source and destination, and it is not shared with any other device.
Yes, of course. That’s what the whole problem is about obviously .
In Syncthing, you could simply try to remove the folder and then re-add it. This should remove the whole thing from the database. When re-adding the folder, I’d also set it to “Receive Only” first just to avoid any potential conflicts. You can change the folder type later if everything is fine after the first scan (which may take a while judging by the size).
@tomasz As I mentioned earlier, removing the folder and re-adding it does not fix the problem, even with a fresh directory. Also, the folder is already “Receive only”.
What if I stop Syncthing, keep my config.xml but nuke the entire index-v0.14.0.db, then restart? Is there anything of persistent value in the database? (There is clearly something of persistent negative value in there!)
Also, I noticed that you’re using GitHub - syndtr/goleveldb: LevelDB key/value database in Go. . Do you know of any tools for interactively poking around inside? Perhaps I can find out what’s gone wrong. If it were sqlite3 I’d already be in there reverse engineering your schema
I have a theory. What happens to data for a folder id when there are other Syncthings cluster members with the same folder id, and they are not currently sharing those folders?
In this case, the folder id is in use on several Syncthing instances, but I disabled sharing because I wanted to make sure I got a consistent copy of one particular “send only” instance, before passing that on to the other members of the cluster. And because I’m dealing with a macOS (HFS+) to Linux (ZFS) send, I wanted to make sure the file name normalisation happened safely before I send everything everywhere. (I have a lot of files with names in many languages.)
@tomasz86 you said that removing the folder should “remove the whole thing from the database” but is it retained if there are other cluster members that it was shared with earlier? Is there some sort of claim on the records from the other members? Could the other cluster members be poisoning my receiver’s database with irrelevant items that don’t exist between the current sender and receiver?
If I come up with a reproduction I’ll let you know.
This rebuilt the database without the poison in it, taking about 20 minutes. The sync with the sender then completed in about 2 minutes. Both ends now show the correct number of files and sensible sizes.
So removing a folder does not clean up the database.
Before all, i’m not an expert and especially not when it comes to the database structure, so please keep this in mind . If you nuke the entire database, the main problem that could happen are file conflicts and/or deleted files getting restored if the state has changed on other devices in between the process. You could mitigate this by setting the folder to Receive Only first, and only then wipe the database out. There are also a few other implications when using one specific non-default config option but that’s likely not applicable here.
As for the other questions, I think a more competent person will need to answer them, as I just don’t know the details when it comes to how the database operates .
.
. If you nuke the entire database, the main problem that could happen are file conflicts and/or deleted files getting restored if the state has changed on other devices in between the process. You could mitigate this by setting the folder to Receive Only first, and only then wipe the database out. There are also a few other implications when using one specific non-default config option but that’s likely not applicable here.
.