This is my PowerShell script. Feel free to utilise if necessary.
$Folderid = "xxxxx-xxxxx"
$Filepath = "xxxxxxxxxxxxxx"
$Apikey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
$Url = "http://0.0.0.0:8384/rest/db/file?folder=$Folderid&file=$Filepath"
$Headers = @{"X-API-KEY" = "$Apikey"}
Invoke-RestMethod -Uri $Url -Headers $Headers -Method Get -Outfile rest.txt
It does not work with Use HTTPS for GUI and GUI Authentication Password though, so I just disable these temporarily when I need to use it.
Thanks tomaz86 - that allowed me to make some progress!
The script worked on W530 which is a Window 10 machine. Output attached: rest-W530.txt (1.0 KB)
It didn’t work on MEDIA-XPS though, which is a Windows 7 machine. Gave the following error:
Never mind, figured out what the issue is … need to upgrade the PowerShell version. Give me a few minutes to do that and I’ll rerun it.
Here you go. Output from both systems attached. XPS is the system with the out of sync items. It says W530 is stuck at 95%.
rest-XPS.txt (1019 Bytes) rest-W530.txt (1.0 KB)
That looks interesting. I don’t yet get it, but you might indeed have hit some obscure corner case bug here.
Two more questions: Which device has an ID starting with WPRAUGR and which with QV5DARG? In one of the screenshots there’s a device “KRIS-S8”: Is it also sharing the folder in question?
Which device has an ID starting with WPRAUGR and which with QV5DARG?
WPRAUGR is KRIS-T530. This is another computer the folder syncs to. I didn’t raise it yet as it was off. It’s on now and MEDIA-XPS says it’s stuck at 95% as well.
QV5DARG is the W530 (KRIS-W530) MEDIA-XPS is YZZTVFE
In one of the screenshots there’s a device “KRIS-S8”: Is it also sharing the folder in question?
No, it doesn’t share this folder. That’s my phone. It shares other folders but not this one.
Correction regarding T530. It’s in sync now so we’re back to the original problem. In case it helps, I’ll restate the problem since it may have gotten a little confusing with all the device names.
MEDIA-XPS says W530 is stuck at 95% and has 106 out of sync items. W530 says everything is fine though. I checked both devices and all the files in question have been deleted and don’t exist on either device.
Screenshot of the remote devices on MEDIA-XPS below:
I suspect W530 and T530 are not connected directly, only via XPS, right?
If you’re interested you can try wrap your head around the following, however I am pretty sure given my “skills” at explaining stuff and that you probably have no prior knowledge on Syncthing internals, it might not tell you much (I still need to put it in writing even if just for my own reference):
The situation is that the XPS never had the file at all. W530 deleted it at "modified": "2020-05-04T19:41:42.3560422-04:00" and T530 at "modified": "2020-05-04T13:49:14.3560422-04:00", without connecting in that timespan. Thus those two are technically conflicts (though it doesn’t matter as both are deletes). The winning side is W530, having deleted first. So XPS thinks T530 is out of date, because it can compare it to W530. However T530 happily thinks it’s up to date, because it doesn’t know about W530 (assuming they really aren’t connected).
The problem here is, that we do not “pull in” a deleted file if we don’t have that file at all. If we did that, XPS would get the same version as W530 and advertise that to T530, at which point T530
(Nice collection of laptops you have there, consider running linux on them, would make them even nicer 
).
I suspect W530 and T530 are not connected directly, only via XPS, right?
Correct, XPS (MEDIA-XPS) is where I store my media (apps, videos, etc). It shares files out to W530, T530, my phone, etc.
The description you gave makes sense I think. I was trying to resolve conflicts in Syncthing and was deleting, moving dirs, etc on W530 and MEDIA-XPS but T530 was powered off at the time.
Do I need to run a -reset-database on MEDIA-XPS at this point?
(Nice collection of laptops you have there, consider running linux on them, would make them even nicer
).
Thanks. 
I like to tinker. There a Linux one in the mix as well but no Syncthing there yet.
No, resetting db on xps wont help at all. The quickest solution is to connect W530 and T530. If you don’t want to keep the connection, you can remove it afterwards and it will still be resolved. However possibly there’s no reason to keep them disconnected at all.
As to how to fix the problem in general: It seems like removing deleted files from the db will do the trick. However we discussed that so many times and there was always problems abound, however I can’t think of what they are now. @calmh @AudriusButkevicius Could you remind me about the error/problem with the following process:
If the local devices plus all known devices have a file deleted, we delete all file infos from db. If all the devices in the cluster removed the file info, the file infos will permanently be gone. If another device hasn’t removed the deleted file info (e.g. because a device connected to it but not us still has an old, undeleted version of the file), we will get that file info again from them on next index exchange (upgrade), but that doesn’t hurt (we don’t even consider it as needed and add an info for ourselves). Essentially this will get rid of file infos truly deleted everywhere, while still protecting against a disconnected devices somewhere at the edge of a cluster overriding the deletion when coming online after ages. Unless of course it wasn’t just disconnected but unshared, and then readded, but that seems like an enough outlandish scenario to ignore it.
The quickest solution is to connect W530 and T530.
By “connect” do you mean have them on or to have them in the remote device list? All of the devices are on now but XPS still says W530 is out of sync … stuck at 95% with 106 out of sync files.
In the remote device list, i.e. directly connected.
The hangup when I did this last time was that we didn’t have a record of the delete time and I didn’t want to drop delete entries that were created recently. (We have that now so I plan to revive this in a while when that has permeated properly.) We even have it twice now that the version number ~= update time.
The other criteria was we didn’t have a need entry for the file… We talked about whether to also require that everyone else was up to date (no older versions in the version list) or not but I don’t remember what we decided. On the one hand it’s a superficial requirement because we can’t know about all devices anyway, on the other hand there might be devices known that are in fact dead since a long time and will never provide the proper delete record… 
That doesn’t matter in my scheme, we can drop them immediately (i.e. on db change) when the last known devices announces having deleted the file.
I definitely wouldn’t drop anything unless everyone is up-to-date (respectively announces nothing for the file, which we consider equivalent right now for deletions). Sure that means we keep deletions indefinitely if there’s one device that hasn’t been removed but never is online, but that’s no concern to me. if that user has a problem with a too large db, they should remove that device already.
In case it helps determine the right course of action development-wise, I’ll try to describe my setup and what I did in more detail.
MEDIA-XPS is the central point for all my media. I created Syncthing shares there and added devices as necessary, so it looks like this:
MEDIA-XPS > W530
MEDIA-XPS > T530
MEDIA-XPS > S8 (my phone)
There is no connection between W530, T530 or S8. That setup seemed to make sense to me as MEDIA-XPS is always on - downloading and serving up media to my house.
What I did: There were Syncthing conflicts I was trying to resolve. I was moving and deleting directories/files on W530 and MEDIA-XPS to try to fix these. T530 was powered off at the time but may have been powered on at some point … I wasn’t doing moves and deletes on it though.
At the end of the whole process, W530 and T530 were fine and indicated everything was up to date but MEDIA-XPS still said W530 was out of date … stuck at 95% with 106 out of sync items.
I’ll try connecting T530 and W530 in a few minutes. I’ll give it a few minutes to see if there’s any additional info you want to figure out what went wrong.
I connected W530 and T530. The problem is still present unfortunately. MEDIA-XPS says W530 is stuck at 95% with 106 out of sync items. W530 says everything is fine.
Screenshot from W530 after adding T530:
Screenshot from MEDIA-XPS:
You also need tho share the folder (the (Unused) shows that it’s not shared).
T530 and W530 both already have the folder shared from MEDIA-XPS. If I share it between W530 and T530 as well, it will look like this:
Is it ok to have the same folder replicated multiple times like that?
When setting up the share between W530 and T530, do I tell T530 to write to the same directory as the share between MEDIA-XPS and T530?
Yes of course, that’s a major point of syncthing - decentralized sync. I’d even suggest doing that in general, not just now.
That seems to have got it! All devices indicate everything is up to date now. Thanks very much for all your help Simon.
