First, thank you to the team who develops and maintains this wonderful FOSS software. Syncthing is truly a wonderful tool in a multi device environment.
I’m hoping for some advice on the following issue:
I run 5 devices connected together with Syncthing in what I would consider a “Hub/Spoke” configuration.
A Fedora 34 Server running Syncthing in a Container acts as the Hub (Home Network)
The following connect as spokes to the Hub:
Win10 Laptop (Work Network)
MX Linux 19 Desktop (Work Network)
Fedora 34 Workstation Desktop (Home Network)
Android Phone running Syncthing for just one folder (hops between Home and Work and Cellular Networks)
All of the “spokes” report through their interfaces (Web, SyncTrayzor, Mobile) that their folders are “Up to Date” with the Hub. When you look at the Hub’s Syncthing web interface it shows all of the spokes are “Out of Sync” with files (usually 1 or 2 files) that are less than a MB. Sometimes the file was a temporary file that was created when LibreOffice was editing the file, sometimes it is a file that was deleted on one of the spokes. All of the files that are noted as “Out of Sync” on each spoke are from a different spoke, and none of the files currently exist on any of the spokes.
Is there anything that I can do to tell the “Hub” that everything is OK?
The files don’t exist on any of the spokes, they are not supposed to exist, they are not files that are being actively edited on one of the spokes. On the Hub’s interface it does say that the folders are “Up to Date” which I believe is true, but where all four of the “Devices” are listed those are all listed as being “Out of Sync”.
This shows all of “Spokes” still trying to sync with the “Hub” 0 B files. They are all stuck at 95%. The two that are disconnected are because they were shut off at the office, but earlier today when those computers were on they were also stuck at 95%.
This final screenshot shows a the file that is out of sync. Notice it is 0 B and it has been deleted off of all of the “Spokes” as it should have been, because I purposely deleted it.
Are the other devices also connected with one another, or only with the “hub”? If it’s the latter, are those files perhaps ignored on any of the other devices? If yes, this could perhaps be https://github.com/syncthing/syncthing/issues/7474. Also, do you use the current or at least a recent version of Syncthing everywhere?
The “Spokes” are not connected to each other. They are only connected to the “Hub” or in this case a server called “tophat”. I have not implemented any ignore patterns on any of my devices. All of the computers connect to all of the folders listed that are called “mowest*” except for the cell phone. It is only syncing one folder so that it has access to a keepass file and some text files that I pass between systems for active notes. As I mentioned all of the files that are “Out of Sync” have been deleted, and I checked they are not on the “Hub” either. I believe that on all of my computers I am running 1.18.1 or newer. Good questions though, thanks for asking for the clarifications.
It seems that something may have been corrupted in the database, but it’s likely impossible to know the culprit after the fact. A non-destructive action to perform would be to remove the affected folders on the affected “spokes”, and then re-add them there at the same paths. Syncthing should re-index the existing data without re-downloading anything.
Of course, I’d always suggest to make a proper backup beforehand.
Ok, @tomasz86, I think I did something wrong. I removed the affected folders on two “spokes”. I stopped “sharing” those folders from the “hub”. Waited a hot second, and then from the “hub” shared those folders again. In their respective interfaces (web for one and mobile for the other) I got notified that the “hub” was sharing a folder with me. I accepted and set up the path to same folder that I had used previously.
Now for the results:
On the android phone after I accepted the folder and connected it to the original folder that I had used previously, all of the same file are there. The “Hub” the Fedora 34 home server interface says everything is “Up to Date” both in the “Folders” section and in the “Remote Devices” section for the Android Phone.
But on the Syncthing mobile interface it says under “Devices” it reads “Syncing (50%)” and has been stuck there even after restarting Syncthing mobile.
A similar issue happened with my home system, with one exception. I got the following error when I tried to reconnect the “Shared” folder from the “Hub” with my home computer running Fedora 34 Workstation, “Error on folder “mowestDocuments” (nztgp-rtvwx): folder marker missing.”
So I looked in /home/mowest/Documents/ and there was no .stfolder directory. So I created one using mkdir and chmod to ensure that it had the same permission as the other .stfolder's on the system. Then Syncthing started indexing the files.
On the “Hub” interface I got the message in the Folders and Remote Devices that everything was “Up to Date”, but look at what the “spoke” interface says.
Now on the “spoke” it sees that folder as “Out of Sync” and is stuck at 66% syncing with the “hub”. Interesting enough, the 11.5 GiB seems to be double of what is actually in that folder which I used du -h /home/mowest/Documents and it said there was 5.8 GiB in the folder.
I’m not sure why the “Hub” seems to be happy with the re-index, but now both of the “spokes” think they are out of sync. This is the opposite of the problem I started with, and appears worse, but I don’t think I have lost any data at this time.
The situation doesn’t look very promising, to be honest, as I myself have experienced similar cases in the past, and all of them eventually ended up in doing full database resets on all devices .
How about trying to do the same thing yet again, this time removing the folders physically too? Basically, you remove them from spokes both in Syncthing and from the storage, then you re-add them and let everything download again.
Just a last comment, but please be very careful with manual recreation of .stfolder. Doing so in an empty path can easily result in mass deletions on the other devices.
Ok, @tomasz86 I have attempted this on my “spokes” of the android cell phone and on my home desktop.
Android Cell Phone: On the “Hub” I first unshared the single folder that I was syncing. On the cell phone I deleted the folder from within the syncthing app, then deleted the folder from the internal storage. On the “Hub” I shared the single folder again. On the cell phone I clicked on the notification that the “Hub” wanted to share a folder, and created a new directory with exactly the same name as the old folder that I deleted. This worked. Everything downloaded again onto the cell phone and every interface on the cell phone and “Hub” show those two devices as “Up to Date”.
The Home Desktop (Fedora 34 Workstation): This did not go as smoothly. On the “Hub” I unshared the mowestdocuments folder. On the desktop I removed in the web interface the mowestdocuments folder. Then using files, I renamed “Documents” to “Documents_Old” so that I had a copy of that data in case everything went south. I created a new folder called “Documents” that was empty of course. Then on the “Hub” I shared mowestdocuments folder again with the desktop. In the desktop web interface I saw the notification that the “Hub” was sharing a folder, I clicked okay to set it up by pointing it to the path for /home/mowest/Documents the new empty folder I created. I soon got the error, “Error on folder “mowestDocuments” (nztgp-rtvwx): folder marker missing.” So that didn’t work.
On the 2nd try I did all of the same above except I created a new folder called “Sync_Documents”. Everything was downloaded from the “Hub” onto the Fedora 34 desktop and everything said, “Up to Date” on both web interfaces. However, I really wanted my “Sync_Documents” folder to be called “Documents”. So I “Paused” syncing both on the “Hub” and on the Desktop for that one folder. I edited the config.xml file changing the name for the folder from “Sync_Documents” to “Documents” and saved config.xml. Then I restarted Syncthing on the Desktop, and “Resumed” that folder syncing on both the Desktop and the “Hub.”
Everything is working fine now, and I have Documents called “Documents”.
It will be a little more hassle on the other machines to do the same, but I will get that done later in a similar fashion, not sure how the above procedure will go for the Win10 computer at the office, but I imagine the Linux Desktop over there will be similar to my home desktop experience.
Just a comment, but you don’t necessarily need to fiddle with config.xml. You can access almost all of the settings using the Advanced Configuration in the Web GUI. This is what I usually do when changing folder paths and such manually. The advantage here is that you’re less likely to mess something up by a typo, and also that Syncthing doesn’t have to be restarted to apply the changes.
@tomasz86 thanks for that encouragement to check out the “Advanced Configuration” in the Web GUI. I never investigated that before. That does enable me to change the folder path. Thanks for the hint. So many wonderful features, what an awesome FOSS project.
.
.