Different exclusion lists issue

Creaserv · February 9, 2018, 9:35am

Hello,

I’m testing a fresh install of nearly 190k files in 20k folders for a tiny 339GiB size.

This folder is on a windows 10 computer.

I connect two others windows 10 computer to this share, each with a different list of exclusion.

The two computers are syncing normally only the files and folders within the exclusion list.

The thing is if I put a new file inside a synced folder then it doesn’t sync back to the root computer.

Example:

Initial Computer with all files (Syncthing in 2 ways sync):

* Folder 1111
* Folder 2222
* Folder 3333

Second computer (Syncthing in 2 ways sync) with exclusion list

!*1111**

So the folder 1111 is synced

Third Computer (Syncthing in 2 ways sync) with exclusion list

!*2222**
!*3333**

So folders 2222 and 3333 are synced on this computer

If i put a new file in the folder 1111 of the first computer then the file is normally synced in the 1111 folder on the second computer. But if I put a new file or modify a file inside the folder 1111 of the second computer then it doesn’t sync back on the first computer.

I tested this behavior with only two computer then the files are synced 2 ways like they should. But with the combination above in 3 computer sync it’s not working.

Am I missing a thing ?

Thanks for any help

calmh · February 9, 2018, 10:35am

I could see a possible scenario that could (but of course shouldn’t) cause this. (Paging @imsodin on the thinking.)

Device B: sees new file, scans it, announces it to A and C

Device C: ignores the new file, sets the invalid bit, announces to A and B

Device A: gets the announcement of the file from B and C, the one from C has the evil bit set, and somehow ends up as the frontmost file in the global list. Device A does not attempt to pull the file because it is deemed invalid.

Maybe this is order dependent or something on A.

We should make sure that given both valid and invalid variants of a file the valid one always wins. I’m not 100% certain that we do.

Creaserv · February 9, 2018, 10:49am

It could be an answer as I see Device A seeing a new folder but not syncing it.

imsodin · February 9, 2018, 12:05pm

That scenario should not be possible: An invalid files should always lose to a valid file with the same version. As file from B announced to A is valid, it should win over the file from C with the same version but invalid bit set.
I checked the code and there is one loophole, but I don’t think it applies to the situation here: When files are in conflict, normal conflict resolution happens (i.e. checking delete flag and mod time, but not invalid), so an invalid file might win over a valid one in that case. I’ll fix that.

Creaserv · February 9, 2018, 12:08pm

The thing is this happens with new files as well as modified files.

wweich · February 9, 2018, 5:09pm

Try pausing device C on device B before adding a new file. If it gets synced, C is preventing it like @calm suggests. If it doesn’t get synced, it has another reason.

Creaserv · February 16, 2018, 11:07am

So I did some tests.

Machine A shares a folder Machine B syncs with exclusion list

Here’s the exclusion list I’m using

!16640*
*

Folders on Machine A are like this

16640 16641 16642 etc…

So with the exclusion list, Machine is only syncing 16640 folder and files.

I put a new file insde 16640 folder on Machine B and then it doesn’t sync back on Machine A.

Syncthing on Machine B doesn’t even see the new file

cosas · February 16, 2018, 11:37am

Did you try to use the Rescan button on B?

Creaserv · February 16, 2018, 11:40am

On both sides it doesn’t change anything.

imsodin · February 16, 2018, 12:06pm

So the problem is locally on just that device. To clarify: Device B has the exact exclusion list you wrote above and you create a file at “16640/newfile” and that does not get scanned (file counter doesn’t increase)? You can enable the scanner debug facility, trigger a scan and check the logs for lines with the name of the file you created.

Creaserv · February 16, 2018, 12:57pm

Yes Device B has that exact exclusion list.

I need to clarify something that I just tested.

It seems it’s just a regular expression issue because I just tested what follows.

Here’s the full folder tree view

You can see that the folders are like this

"AFFAIRES 2016"
|- "16628 NAME OF THE PROJECT"
|- "16629 NAME OF THE PROJECT"

So the exclusion list I wrote before syncs on Device B the following treeview

"AFFAIRES 2016"
|- "16640 NAME OF THE PROJECT"

So I tested to copy on Device A the 16640 folder at root so now my treeview on Device B is this:

"16640 NAME OF THE PROJECT"
"AFFAIRES 2016"
|- "16640 NAME OF THE PROJECT"

Guess what ? When I put a new file on Device B inside the first 16640 folders then it syncs back normally to Device A.

As you asked @imsodin I enabled the scanner debug and it seems that it’s ignoring folder “AFFAIRES 2016” because of the exclusion list.

So I’m kind of stucked because if I modifiy the exclusion list like this I think it should work

!*2016**
!16640*
*

But I don’t want all the folders from “AFFAIRES 2016”. I tested like this

!*2016/16640**
*

But it’s just ignoring “AFFAIRES 2016” folder again

imsodin · February 16, 2018, 1:14pm

* matches “Affairs 2016” and thus excludes it, so its child files are never matched. You need to specify ignores for every level e.g. like this:

!*2016/16640*
*2016/*
!*2016
*

Creaserv · February 16, 2018, 1:26pm

ok so it’s working but it’s kind of tricky because when you see it syncing one way you think it should sync the other way too but in fact it’s not.

I modified like this so that it’s working for each folder

Thanks @imsodin

PS: Is it the default behavior not to delete local files that are exclude from exclusion list if we modify the exclusion list ?

imsodin · February 16, 2018, 1:43pm

In that exclusion list * has no effect because !* is the opposite of *. I assume what you want to do is:

!/*/16640*
/*/*
!/*
*

The leading / anchors the pattern to the folder root.

Sorry, you lost me there.

Creaserv · February 16, 2018, 1:50pm

I can confirm that both your exclusion list and mine are producing the same effect.

If I remove !/*/16640* from the exclusion list on Device B then it won’t sync again this folder but the files will remain on Device B even if it’s now exclude from syncing. What I wanted to know was if it was the expected behavior or if syncthing should delete the files on Device B (which is in fact what I want to achieve)

imsodin · February 16, 2018, 2:03pm

That’s expected, Syncthing definitely shouldn’t delete anything that’s ignored (except with (?d)).

cosas · February 16, 2018, 2:11pm

This is in exclusions list as an exclusion from exclusions, so it is an inclusion. Removing an inclusion, as far as it is not re-introduced lower in the list turns it (as included in final general * exclusion) to be excluded. Then ST just ignores matching items. Beside of this we hope st won’t deleted (nor sync) anything on its own, namely won’t delete (nor sync) anything if it is not instructed to do so from a remote device. From a local POV, adding or removing ignore pattern won’t be itself the reason for deletions. The reason would be missing files/dirs on the other side once an exclusion is dropped.

How do I do :

I include locally

1a) I work on files and let sync

I remove inclusion locally
I delete locally what I don’t need locally anymore.

In you case your 1640 thing will match the general exclusion, so will be ignored. ST won’t sync local (nor remote) 1640 so you’ll need to manually delete it locally if you don’t need it. Deletion won’t be sync’d

Creaserv · February 16, 2018, 2:30pm

That’s what I’ve been doing as well

cosas · February 16, 2018, 2:45pm

Strange wording then. I read you want to achieve ST to delete the files once ignored !

Creaserv · February 16, 2018, 4:31pm

That’s right. I wanted ST to delete the ignored files that were previously synced. That’s impossible so I do it manually