Hi all,
I’m noticing a host that claims Out of Sync Items :
When i look at the filelist, it’s clear that those are files which were deleted on the host. Checking the remote device, the files are indeed still present instead of having been moved into the versioning folder. Also, the remote device claims that everything is good and Up to Date :
I have turned on DB and MODEL debugging facilities on both hosts. Will post logs asap. What other steps should I take to get to the bottom of this ?
First check that you didn’t set “ignore deletes” and forget about it.
I swear there are no IgnoreDeletes activated. This is a setup that has been working solidly for well over 2 years… no config changes.
Every day around 6am, there is a scheduled mass-deletion event, so i should get decent debug logging tomorrow morning.
Interesting, I also have one machine stuck like that: 6 items and also at 95% with 0 Bytes - coincidence?
I noticed that in modal “out of sync items” there a 3 items with “0 B” and 3 items with “” (empty string) as size. Modification times are more than one month ago. Also there is an empty .stignore (created July 25th) on receiving side although no ignores are used. Source is set to send only (possible cause in combination with empty .stignore?) Everything else (add, delete, modify) works fine for other files.
I will check originating side for more details.
I also have these 0 Byte files in my Syncthing instances. I believe that these files are created (0 Byte), Syncthing pics them up, then the files are filled with data and deleted again before Syncthing can finish the scan of the file.
I was never able to reliably test it, so I didn’t create a bug report…
I am also interested if that happens. In my fully scripted system I had occurrences of devices being stuck at 95%, but i could not detect exactly which files and what the reason was.
Update from source side:
The folder is “send only” and does not use ignores or file watcher. The files are still there (not deleted) and have more than 0 bytes (all below 100 KB).
@bigbear2nd, @schneemensch: are you also using “send only” folders where this happens?
Yes, I am using a sendonly folder as well. I cannot tell more about the files, because the sync happens over night and is automatically unshared afterwards.
in my case, it is also a send-only origin. But the actual issue is that on the original host the files are deleted, and the deletion never propagates to other side of sync.
@calmh: the overnight logging is huge, 300megs, how can i get that to you guys ?
I don’t use the send only option, I use “ignore delete”.
I just saw @calmh answer about the ignore delete.
The phone is creating the data (pictures).
The nas is storing the data and has set “ignore deletes”
The nas is the device with the 0 Byte files. I believe the issue should not arise on the nas.
I see there are also ignore patterns. Make sure those match your expectations and are the same on both side.
@Tom I see logs and stuff in PM, however no time/inclination to look through that at the moment. If there are errors, look at those first.
@calmh understood.
Aug 27 09:09:17 backup-nas syncthing[19139]: [JKZ4T] 2018/08/27 09:09:17.838298 logfs.go:61: DEBUG: mtimefs.go:70 basic /srv/dev-disk-by-id-md-name-nas-softraid/syncthing/kate/VILV-DB-BACKUP/ Lstat DBA_backup_2018_08_23_063002_1104513.bak {0xc4239aaea0} <nil>
This is a repeating msg regarding one of the files that never gets deleted. Well, in this case the file is supposed to be moved to .stversions by Staggered Versioning.
That’s an entirely normal thing to be happening. Almost all debug level things are; they just bring developer context to any errors, where the errors should be visible anyhow.
unfortunately i’m not seeing errors… just the symptoms, a file deleted on one side of the sync never gets moved into versioning.
[TXZZI] 2018/08/28 11:33:10.202261 leveldb_dbinstance.go:459: DEBUG: need folder="vnyi3-dsvsk" device=JKZ4TOI-VTZPV4T-7FBFGD4-BZ2KPWW-3XCSH23-YNCRCTO-N44WZIZ-DFM6BQ4 name="DBA_backup_2018_08_23_063002_1104513.bak" have=true invalid=false haveV={[{TXZZIYN 1}]} globalV={[{TXZZIYN 2}]} globalDev=7777777-777777N-7777777-777777N-7777777-777777N-7777777-77777Q4
still not an error, does it look a little funky though ?
They, JKZ4TOI, lack an update that you (7777777, the “local device”) have, that TXZZIYN did. Not funky as such.
well, JKZ4TOI should version the file DBA_backup_2018_08_23_063002_1104513.bak that was deleted at TXZZIYN.
Sure. To find out why that didn’t happen you need to look for errors on JKZ4TOI. The above log line just says that they haven’t sent an update about handling the file, which you already knew.
Look, I’m trying to convey that reading random debug output like tea leaves isn’t useful. The information is dense and hard to interpret.
Assuming there are no errors anywhere JKZ4TOI, that is where you might want to enable model tracing. Or, use the /rest/db/file endpoint to look at what it knows about a file that should be deleted but isn’t. In fact, I’d start there. Lets see what it thinks about one of those things that are not deleted. The config for JKZ4TOI would also be good.
so i ran the folder & filename against the rest api,
this is the response from the host that will not move the file into versioning - i.e. JKZ4TOI
https://pastebin.com/bB3rs9va
And this is what TXZZIYN thinks it knows about the same file.
https://pastebin.com/Hg7dysXU
Does this look like JKZ4TOI hasn’t heard that the file is deleted ??
ps : sorry for using pastebin, the forum seems to mangle the JSON copy/paste ?