Unintended deletion were on the devices that don’t have !ignores. One of them is up 24/24.
I work this way : I edit files in say 2006/2006 07 foo in the device having the !ignores. I wait/check for sync to the 24h up device, then I remove the !200604 07 foo line out of the .stignore file with the GUI, then I delete the local ..2006/2006 07 foo directory.
I think the data loss happened on the 24 up device, as I found the files back in the staggered trash can with time stamps before the second device was fired up.
Same thing happened in other directories I recently worked on.
Thank you Audrius. Please when was this change introduced ? 47? 48? 49? rc?
Was there anything changed in the documentation about this, or was it/still is it undocumented ? I just had look but was unable to remember old doc if ever.
I reproduced observed the out of sync issue and know how it happens - it is due to the new need database stuff in combination with ignored files, but not related to descending to catch include paths. It’s totally possible that what you describe in your first post is unrelated to this (haven’t found anything about that yet).
Everything files/dirs/shares/!ignores yet existed at the moment ST was upgraded on A (the !ignoring device) was upgraded to 0.14.49-rc1. The 2 other devices in the cluster are/was set to autoupgrade to stable releases only. At the time of upgrade, the cluster was UpToDate on the 3 everywhere.
I can’t remember if I deleted 200604 07 foo then the related !ignore line before or after the upgrade.
Like you I didn’t deleted the 2006 directory as I still have the !200604 17-23 bar to work on inside.
Is this is the new best syntax for this goal?:
!/2006/200604 07 foo
!/2006/200604 17-23 bar
!/2014/201402 foo
*
Notice the leading / I added to feel secure
Also, the strange thing, before & after I reset-database on the 3 nodes is that B & C show(resurects) missing OutOfSync items that where deleted several months ago.
See Deleted files remain ‘Syncing (95%, 0B)’ for report of resurrected ‘zombie’ files after upgrade to .49rc. I was experimenting with ignores on separate folder when zombies (deleted files from days ago) appeared as out-of-sync. Reverting to .48 made no difference. Resolved by --reset-deltas on offending machine and deleting ignores. Updated again to .49rc and no further problem but yet to reintroduce ignores. I might sit on my hands for a while.
[EDIT] other things: I found fswatcher was enabled on dev C with default max_user_watches (8192) which was roughly 1/4 the number of files in all the shares on this dev. I disabled and set MUW to 131072 for further re-enablement.
And yes, those ignore patterns should work with 0.14.49.
As to reproducing the deletion issue: I don’t see anything to try/change on my reproducer based on your answer (or I didn’t understand). If you can reproduce it, please post steps the way I did. As it stands there is nothing more that I can investigate about this for me.
I’ll do my best Audrius, but it will be very difficult as I didn’t change the way I usually work with the files in this shared folder. As I said, given the !ignored syntax I used while on v48, maybe the introduction of new behaviour in 49rc did something, maybe plus a bad setting in one 48stable about fsWatcher. Now I changed it on my 49rc device.
Maybe also related to #5007 : device A is filling its own Recent Changes GUI list each minute, as well as the 2 other devices one, with:
Device
Action
Type
Folder
Path
Time
DevA
modified
dir
Photos
2012
2018-06-17 15:46:55
DevA
modified
dir
Photos
2012
2018-06-17 15:46:08
DevA
modified
dir
Photos
2012
2018-06-17 15:45:11
DevA
modified
dir
Photos
2012
2018-06-17 15:44:18
DevA
modified
dir
Photos
2012
2018-06-17 15:43:12
Although all 3 devices show UpToDate, B & C blink each minute showing Synching 0% (intended as fsWatcher is disabled), and sometimes briefly Out Of Sync 99% missing 1 item for 128B.
[EDIT] I commented out all !ignores but one (!/2012/201208 foo … and * of course )
… and went on my usual work by treating my local .../2009/200912 26 etc end of year. The remaining job was to look at the pictures inside, decide which ones to keep and delete the others. Before I started I checked the Up to Date status on the 3 devices (OK) and I cleaned .stversions/2009 everywhere to track any further possible issue.
I did the job, then locally added added my usual flag inside the subdir (a 0 byte empty file called 0k), then I was ready to remove the !ignore and delete the local subdir (and even the no more needed 2009 dir as there is the single 200912blah dir inside) after I check everything is still UptoDate … but:
Now, devices B & C show syncthing 99% ~265MiB to Device A for 107 items! Local subdir is 123 items ~297MB, remote ones (B & C) are 153 items ~382MB, and remote .stversions/2009 are only 13 files ~20MB when I deleted 43 (153+13-123) on dev A.
To have better/cleaner view I re-commented !2012/... so that only one !ignore remains, the 2009 one.
On ALocal State shows 123 files/1 dir (what maybe ok or ko, depending we count 2009 for one, and 200912 26... for an other)
While A is scanning, Global State dir number decreases by one (1) and increases back by 1 in the end, on every scan even no real change happened.
The 107 files B & C would want to send to A either are & were yet on A, or are neither in A nor in B/C’s .stversions .
Either initial Up to Date was wrong (actually it wasn’t, I checked this) or st was puzzled at some point during the deletions on devA.