Can you provide screenshots of both syncthing UIs expanding the problematic folder?
On one of the Linux boxes:
Same number of files in the Global State, Local State, etc.
Well it’s not the same number of folders yet both of them think locally they are up to date, so I suspect the ignores are different on the other device causing this.
The difference in folders the number of folders is that damn Garmin one that the mac is trying unsuccessfully to sync to the Linux boxes. The .stignore file on the Mac is
(?d).DS_Store
Photos\ Library.photoslibrary
and on the Linux boxes is just
(?d).DS_Store
because the Photos\ Library.photoslibrary directory lives on the Mac. Both those files were last modified last June, and everything was working fine until two days ago.
The difference in this case of the “95% problem” is that the dir actually exists on one device but the others don’t think they need it - so it’s not “just cosmetic”. Can you give us the output of
curl -X GET -H "X-API-Key: *yourapikey*" 'http://localhost:8384/rest/db/file?folder=*folderID*&file=Garmin' on both sides (replacing … with the appropriate values).
On an unrelated note: What’s the backslash in your ignore patterns meant to do? If it’s supposed to escape the whitespace, that’s not only unnecessary, it won’t work: the backslash will be interpreted literally (subject to error, but I am quite sure 
).
I get “Not authorized” on all three hosts. If I put the admin page userid and password into the URL, I get “CSRF Error”.
Authorization shouldn’t be a problem, that’s handled via the api key. If you have https for the GUI enabled, you need to add -k to the curl command and use https://.
It’s not your UserID or Password, that is needed.
In the WebGUI, you need to go to
Actions -> Settings -> API Key
There, copy the Key mentioned there onto *yourapikey*
EDIT: Also check, that the API-Key fits the IP-Address/Hostname, you enter after HTTPS…
Ok, on the Mac I get
{
"local" : {
"type" : 1,
"numBlocks" : 0,
"deleted" : false,
"sequence" : 159733,
"size" : 0,
"noPermissions" : false,
"modifiedBy" : "CAX3IX6",
"permissions" : "0755",
"version" : [
"ARMRRQT:3",
"CAX3IX6:3",
"VZFF7W2:1"
],
"name" : "Garmin",
"invalid" : false,
"modified" : "2018-03-20T17:24:08-04:00"
},
"global" : {
"noPermissions" : false,
"size" : 0,
"sequence" : 159733,
"deleted" : false,
"numBlocks" : 0,
"type" : 1,
"modified" : "2018-03-20T17:24:08-04:00",
"version" : [
"ARMRRQT:3",
"CAX3IX6:3",
"VZFF7W2:1"
],
"invalid" : false,
"name" : "Garmin",
"permissions" : "0755",
"modifiedBy" : "CAX3IX6"
},
"availability" : null
}
And one of the Linux boxes:
{
"global" : {
"numBlocks" : 0,
"size" : 0,
"deleted" : true,
"noPermissions" : false,
"name" : "Garmin",
"invalid" : false,
"modifiedBy" : "VZFF7W2",
"version" : [
"ARMRRQT:3",
"VZFF7W2:1"
],
"sequence" : 108534,
"permissions" : "0",
"type" : 1,
"modified" : "2018-03-19T20:49:54.892386217-04:00"
},
"availability" : [
{
"fromTemporary" : false,
"id" : "ARMRRQT-..."
}
],
"local" : {
"name" : "Garmin",
"noPermissions" : false,
"deleted" : true,
"size" : 0,
"numBlocks" : 0,
"version" : [
"ARMRRQT:3",
"VZFF7W2:1"
],
"modifiedBy" : "VZFF7W2",
"invalid" : false,
"permissions" : "0",
"sequence" : 48451,
"type" : 1,
"modified" : "2018-03-19T20:49:54.892386217-04:00"
}
}
The linux device simply doesn’t know that the mac device has an updated “Garmin”, i.e. mac does/did not send the appropriate index update. Running mac with -reset-deltas should fix it, but you already did that with no success. Can you enable both model and db debug facilities on the linux side and capture those logs, and start Syncthing on mac again with -reset-deltas. Then see if there are lines containing “Index (in): *mac-device-id*” or “Indexupdate (in): *mac-device-id*” and any lines containing “Garmin”.
And just to be sure, can you confirm that CAX3IX6 = mac, VZFF7W2 = linux box and ARMRRQT = some third device that didn’t feature in the rest calls/screenshots.
While I wait for the Linux box to catch up, I have another question. One of the Linux boxes (ARMRRQT) didn’t have a API key, and when I told it to generate one it didn’t “stick” - it was gone when I re-opened the drop-down.
No mention of “Index”, “Indexupdate” or “Garmin” on the Linux side (VZFF7W2) logs. Only a few mentions of the Pictures directory by name. The Mac is still saying “(99%, 128B)”.
Any mentions of the mac device id at all?
That doesn’t sound right indeed - check whether there is an API key in the config.xml. However I would open a new thread if this requires more investigation, otherwise things get messy.
Almost all the mentions of the Mac are about the src directory, which is reporting 100% on all devices.
I’ve stuck the whole log file in https://pastebin.com/WRpaHiPL
The grep output is definitely not complete, there should at least be messages about the connection like the ones you posted earlier. Most likely the web UI has a finite buffer and the huge load from model&db debug fills it up. So please get the logs from your logfile or system logging service.
Looks like the logging went to /var/log/daemon.log. It’s freaking huge so pastebin won’t cut it. I’ll upload the results of grep -e 'Index (in)' -e 'Indexupdate (in)' -e 'Garmin' daemon.log to Dropbox.
(BTW: The one I couldn’t set an API key on? It has an API key in the config.xml I was able to use in that curl command. Since it’s on the LAN with the Mac, I’ll probably use it for future testing.)
The initial index is arriving:
Mar 21 19:13:18 backup syncthing[676]: [VZFF7] 2018/03/21 19:13:18.840004 model.go:842: DEBUG: Index (in): CAX3IX6-FRBEB53-FUQY2EE-G5U4S7Z-VLMSV3B-JERGFIG-PUIF3JE-WZSFSQA / "Pictures": 267 files
However “Garmin” isn’t in there (there would be a insert;.*Garmin line). Looking at the code it should be impossible to get a result from rest/db/file for “Garmin” and it not being in the sent index. To be honest I have no clue what’s going on. Full logs from both sides may have clues in it, so if you could upload them that’s a first step. Otherwise maybe @AudriusButkevicius or @calmh have specific ideas - my next step would be creating a debug binary with lots of debug logging spammed in there.
Makes no sense to me. The directory was deleted at one point, but is currently resurrected on the Mac (CAX3IX6). This doesn’t seem to propagate. No idea why not.
Well, if you can find anything in these giant dumps, here are the full Mac logs (running with STTRACE=db,model ~/bin/syncthing -reset-deltas -logfile="syncthing_mac.log" --logflags="3"
And the full Linux logs from earlier: