Did something recently changed in ignores handling ?


(Audrius Butkevicius) #21

Sorry, but I have no idea what you are trying to describe here. What we care is a reproducible list of steps on an empty folder like @imsodin has provided, that produces some issue.

Everything else you provide has very little value, as it’s it only has meaning to the state of folders you have, not the state of folders we can reproduce.


(Russell) #22

@imsodin On .49rc, if I introduce an ignore pattern for a .~lock file on machine A, it works and excludes the lock file as expected on other nodes. A similar ignore pattern on machine B causes machine A to appear Out-of-Sync after the .~lock file is deleted by closing the open file on B.

Removing the ignore pattern on A makes no difference. Remove it on B and Out-of-Sync folder is now Up-to-Date on A.

Is this what you were referring to earlier in this thread when you said rc2 would address this?


(Simon) #23

Not sure, but likely - I suggest you update to rc.2 and try again.


(Russell) #24

Updated to .49rc2 and the issue is resolved.

Thanks @imsodin


#25

For me too


#26

Not that much.

I manually dealt with the !2009... problem, then removed the related line.

After that I uncommented !/2012/201208 18-24 foo and only this one.

devA (where !ignore is): folder and DevB & devC are shown Up to Date

Global State 31 491 551 ~82 GiB
Local State 138 1 ~316 MiB

devB and devC (no ignore, no !ignore). Folder is shown Up to Date most of the time and briefly blinks Out of Sync 1 item 128B sometimes (not every minute as I previously said, but always just after the minutely scan), but devA is showing stable Syncing (99% 316.45 MiB) Out of Sync Items 138 items, ~316,45 MiB

Global State 31 491 551 ~82 GiB
Local State 31 491 551 ~82 GiB

More info : fsWatcher is disabled on all 3 dev for this folder, scan schedule is set to defaults, and the 138 files are the same everywhere (sha256sum checked). Adding/editing/deleting new files either on devA or DevB finely sync (Out of Sync items remains to 138). Recent Changes list is filled on each device by this very same line every minute:

devA modified dir Photos 2012 timestamp

Additional info, maybe related: the out of sync files are pictures I previously edited with exiftool (retagging EXIF date data then batch renaming), using the following parameters: -overwrite_original_in_place -P to preserve fs date attributes.


(Audrius Butkevicius) #27

As I said, this is not very valuable, unless you produce steps on how to reproduce this in isolation.


#28

Hi I didn’t read well Simon’s advice and kept on digging the issue I meet. Deletions are difficult to reproduce, although I got something I describe below because it took me a big time to redact and want to keep it for further reference. Then I’ll post another message in this thread describing what happens in Simon’s scenario.

What I meet is the following: until 0.14.49-rc1/rc2 were published I used to work as described above. Now my devA is 0.49-rc2 and devB is still 0.14.48stable

So I created an empty subdir in devB, then in devA I replaced my ignores:

...
!/2012/201208 18-24 foo
...
*

by:

...
!/2118/211801 01-*
//!/2012/201208 18-24 foo
...
*

Notice the files were yet synced OK before 0.14.49rc so I suspect something wrong in 0.14.49rc

devB Recent Changes

Device Action Type Folder Path Time
devA modified dir Photos 2012 2018-06-21 19:33:13
devA modified dir Photos 2012 2018-06-21 19:32:06
devB deleted dir Photos 2118/New folder 2018-06-21 19:32:03
devB added dir Photos 2118/211801 01-02 barfoo 2018-06-21 19:32:01
devB added dir Photos 2118/New folder 2018-06-21 19:31:08
devA modified dir Photos 2012 2018-06-21 19:31:01
devB deleted dir Photos New folder 2018-06-21 19:30:01
devB added dir Photos 2118 2018-06-21 19:29:59
devA modified dir Photos 2012 2018-06-21 19:29:57
devA modified dir Photos 2012 2018-06-21 19:28:55
devB modified dir Photos New folder 2018-06-21 19:28:52
devA modified dir Photos 2012 2018-06-21 19:27:49
devA modified dir Photos 2012 2018-06-21 19:26:36
devA modified dir Photos 2012 2018-06-21 19:25:22
devA modified dir Photos 2012 2018-06-21 19:24:14
devA modified dir Photos 2012 2018-06-21 19:23:05

devA Recent Changes:

Device Action Type Folder Path Time
devA deleted dir Photos 2118/211801 01-02 barfoo 2018-06-21 19:34:02
devA modified dir Photos 2012 2018-06-21 19:33:12
devA modified dir Photos 2012 2018-06-21 19:32:06
devA modified dir Photos 2012 2018-06-21 19:31:00
devA modified dir Photos 2012 2018-06-21 19:29:56
devA modified dir Photos 2012 2018-06-21 19:28:55
devA modified dir Photos 2012 2018-06-21 19:27:49

What I meet after I set a single ignore all (*) in devA, reset-database on devB then on devB: >>> all UpToDate.

Then I release my !ignore in devA for a sub-subdir >>> devB reports all files in the sub-subdir missing (OutOfSync a hundred of files) in devA although they really are there. devA still shows UpToDate. Renaming a file (let’s call it file) on devB to fileR triggers the following on next sync:

fileR is created in devA…but file remains there in devA! Then file is created (synced) back to devB (like if ~room~ was made for file by renaming file to fileR). Now deleting file either on devA or devB deletes it on the other side :slight_smile: . Now, renaming fileR back to file either in devA or devB renames it fine on the other side.

After batch treatment of the ~100 files, only one item remains OoS from devB PoV for 128B : the 2012 directory, filling the Recent Changes list on both devices stating devA modified it, even after I commented the !ignore in devA. I got rid of it by unsharing the folder to devA in devB, and to devB in devA, then shared back and left the !ignore commented (I could have deleted it).

Ok. Let’s keep that in a corner of my neuron and go to Simon’s scenario.


(Audrius Butkevicius) #29

I am sorry, but nobody is going to try and understand these long posts.

If you want someone to look into this, produce a repeatable set of steps someone has to follow in a clean empty folder (not subfolder) to reproduce the issue.

As in.

  1. Create folder on A
  2. Create folder on B
  3. Ignore foo/* on A
  4. Create file foo/bar on A

etc, etc etc.


#30

Not that long : only the log makes it long.


(Audrius Butkevicius) #31

It’s not in any reasonable format anyone can consume, comprehend and reproduce this. It’s heavily biast to some unknown folder state that you have, that potentially nobody can reproduce. If you wish someone to look at this, please provide what is being asked, which is a repro in a new folder with step by step guide.

If you don’t care about this, then there is not much value in you producing these posts as we have no way of reproducing the same state you are in, hence I doubt anyone is going to look at this.


#32

@imsodin & @AudriusButkevicius

I just carefully followed the steps from post #8 and I get the same results : local state on A drops to “1” and nothing is deleted on B.


#33

I think this is what I meet, as long as un-ignore is synonym with !ignore what was //!ignore just before (ignore being the relative path to a sub-subdir - of course followed by the legacy syntax (dir/*/n !dir/n *) to which I switched back to feel like taming a bit the issue):

When I do this in devA (49rc2) nothing happens here but a scan, GUI is UptoDate and no conflict files are created. On devB (v48 no ignores and RW) this makes all files in subdir to show as OutOfSync items. No conflict ATM.

How I deal with this:

Batch rename all files on devB, with an easy-to-remove suffix. => suffixed file are quickly created on devA, help with blocks re-use, and soon original filenames are replaced by sync-conflict-timestamp-devB.ext, then these conflicts are quickly synced back to devB. Delete conflict and rename back, both either on devA or devB, then I’m done.

Note that doing the first rename on devA doesn’t create conficts and leaves the originals on devB. I think I could delete them in their now single place devB, then rename back anywhere.


(Simon) #34

I am sorry, I do not really understand what you are doing and why (and since you cited me I feel compelled to react :wink: ). Yes, if you have a dir ignored on devA and change files in there while ignoring, then as soon as you stop ignoring (either by removing the matching ignore pattern or add one with the ! prefix), conflict copies will be created. However this should only happen if the files actually differ (being deleted does not count as different, deletions always lose an conflict). So in the end I am not sure whether you encountered a problem and if so, what was it?


#35

Sorry for late reply:

The devB with all files only uses stable releases. The devA that uses ignore patterns follows rc releases.

Selected files having been synced OK at some point in the past thanks to a bunch of !ignores, then discoupled just after r48stable was released/installed on each end by commenting !ignores this way: //!ignores. Now devA being upgraded to r49rc, when I ~free/release~ an //!ignore removing the // nothing happens but devB showing OutOfSync items to devA forever in its (devB) devices list.

To tell it short, “nothing happens but…” means the devB hosting all files shows devA with OutOfSync for freed/released files (on devA) and no conflicts are created. In other words, as long as files !ignored then //!ignored then !ignored again should conflict is intended, the problem is that nothing tell us the reason of OoS in the GUI and no creation of conflict files in the filesystem can give us a clue of what happened. Should I add that devA remains UpToDate.

Renaming files on devB, e.g. mv file1.ext file1renamed.ext, then immediately file1renamed.ext is synced to devA, and file1-syncthing-conflit-timestamp.ext are created on one side then synched to the other. I think I told above on which side conflict files are created first then synced back to the other. IIRC, devA->devB


(Simon) #36

I finally managed to follow, see https://github.com/syncthing/syncthing/issues/5053

Thanks for reporting!
Please try to keep future reports focused on one thing - listing steps and spelling out what happened compared to what you expect to happen are great for that. It makes it so much easier to understand, where the problem lies.


#37

Thank you for efforts Simon.

Although, I must say that some steps in your “Steps to reproduce:” aren’t required in my case for the issue to happen (although after devA upgraded to 49rc4, yesterday I unignored 1 subdir and tonight I unignored 2 other subdirs and the issue didn’t happen again).

The unrequired steps are:

1.Share a folder between devices A and B.

2. Modify file F on A, wait for sync.

3. Modify file F on B, (of course wait for sync is required ATM).

4.Ignore F on A.

5.Edit F on A and B.

4bis. Upgrade A from 48 to 49rc(<4 ?)

6.Unignore F on A.

Expected: ST keeps quiet

Actual: Nothing is (and nothing would need to be) synced and no conflicts are created, A GUI shows all UtD, B GUI shows A misses all files.

sha256sum on each end confirms files are the same