I have some folder on a Windows machine where I found that random files and complete sub-folders were not synced to the two other Debian machines. Initially, while those Debian machines were still on 1.19, I found that not even any updates to the missing files caused them to be re-synced, but this problem is gone by now. I think it’s not related to the Debian machines actually, but maybe to a restart of the Windows machine.
The core issue remains though: Several files and sub-folders of that specific folder are still missing on the other Syncthing instances. If I modify a single file in a sub-folder, it suddenly gets synced to those other instances. However, simply forcing a rescan of the entire Syncthing folder is not enough. So I have a feeling that Syncthing “sees” that the sub-folder contains no changes (does it use the NTFS journal or object ID changes?). I should note that this directory structure is a copy of another directory structure within the same sync folder, and the other copy of the same directory synced without issues.
I noticed that a tool called stindex was mentioned in many related forum posts, but the ones I could find were several years old and the attached stindex version cannot seem to handle the current database. Where can I find a current version of stindex to confirm whether the database has any issues?
It’s possible that you’ve hit one of the index exchange bugs that were present in older versions. Initially, I would recommend that you start one of the machines with the command line option syncthing --reset-deltas and see if that solves the problem.
Not really, but syncthing does indeed not rescan files if neither the file size nor modification time (or other relevant metadata) has changed. Generally, a rescan does nothing if nothing has changed. In syncthing, each device pulls the data that it thinks it needs. If a device doesn’t download something, that’s probably because it thinks that it already has everything (which may indicate an index exchange bug being present).
Unfortunately, running syncthing --reset-deltas on one of the Debian machines didn’t change anything. I then ran it on the Windows machine to be sure, and that also didn’t cause the files to appear on the other machines.
Okay, so something else is off then. Is the Windows folder on an unusual filesystem (which on Windows is pretty much anything but NTFS) or a network share?
I also encountered this problem recently. I downgraded Syncthing on Windows to v1.27.12, and the problem disappeared.
You can also try it, to see if it was caused by some bug in v1.28.0.
Since the missing files are not critical at this point in time, with the knowledge there might be a bug in v1.28.0 I would be willing to keep that directory in the state it currently is, in case Syncthing developers need me to test something to figure out what’s going on.
I think I have the same problem. A few days after recently updating to a new Syncthing version, I noticed a specific folder did not synchronise anymore. This folder is not in the ignore list (and no settings were changed compared to the working state before).
Syncthing claims all shared folders are up to date and doesn’t perform any sync actions when I add new files below the ‘broken’ folder.
v1.28.0, x86, Linux Mint (PC)
to
v1.28.0, ARM, Raspbian (RPi 5)
To supplement the information in the previous ticket:
Going by the modified dates on the files in the source folder missing from the destination folder, synchronisation stopped somewhere between 2024-10-18 and 2024-11-05.
Syncthing on the PC was updated on 2024-10-20 (1.27.12 → 1.28.0).
Syncthing on the RPi was updated on 2024-10-30 (1.27.12 → 1.28.0).
Since no syncthing settings were changed on either device in that period, I strongly suspect the update triggered or caused this change in behaviour.
It’s not at all clear to me what’s going on here. Would you be willing to try starting Syncthing with the --reset-database command-line option? This will reset the database and force a full rescan and resync — it may take a while. You may choose to do that on all Syncthing devices with an issue.
I reset the database on one of the Linux machines. If it doesn’t help, I will try the same on the Windows source machine tomorrow, but it will take a while.
No change after resetting the Linux machine. It really seems to be an issue with the source machine of those files, not the machines that are being synced to. So I will reset the Windows machine next, as that’s where the files should be coming from.
The global and local numbers were consistent before the forced resync. Now SyncThing apparently became blind to not just that one directory, but several other directories too, resulting in about 30GB worth of data not being synced anymore.
If nobody smarter/more knowledgeable than me has a better idea, I would probably do the following:
Confirm that I fully understood the ignore patterns and make sure they weren’t part of this issue
Go through logs with debugs enabled one at a time. I mean enable one debug facility, either restart Syncthing and wait an hour, or just wait an hour, examine the log, and if nothing found look to the next one. I would suggest config (briefly, if something is wrong here I suspect it would show up on startup of Syncthing), connections, events, fs, protocol and scanner.
And if that didn’t get me anywhere, I think I would uninstall Syncthing everywhere, reboot everything using Syncthing, and reinstall from scratch.
I can confirm that my ignore patterns shouldn’t match any of the missing files that I checked. There are quite a few patterns but they are all pretty specific, and I have been using them for years. So next up will be enabling debug logs.
I would also like to ask for full screenshots of the Web GUI from both sides, not just cut-out parts of it. It should be easier to understand the full synchronisation state that way.
Okay, I solved the problem, but not in a satisfactory way - because I still don’t know what the root cause is: I enabled the fs debug logs, and I saw that Syncthing suddenly just plain stopped logging anything after it encountered a junction folder - specifically a junction folder inside the original, troublesome folder from my first post. After I deleted and recreated the junction, the problem is completely gone, and everything is being scanned again.
Now, the weird thing is that I have several of these junction folders in the same Syncthing folder, all pointing to the same destination (essentially just a way to save some space in some otherwise identical source code repositories). So far they have never caused any trouble.
Here’s one hypothesis of what could have happened: When I created this folder, Syncthing might have already started syncing and might have been syncing files inside the folder that was turned into a junction just seconds later. Maybe this caused some inconsistency. However, it wouldn’t explain why it was still broken after passing --reset-database.
Junctions don’t work correctly in the current version. This is a known issue which has already been fixed, with the fix available in the current pre-release. It is going to be included in the next stable version planned to be released on December 3. See https://github.com/syncthing/syncthing/issues/9775 for details about the issue.
