Ignore pattern modification and override changes

Support
4

Overriding does not delete anything, it just forces everyone to think that the files you have are newer. Since you technically don’t have the ignored files, they are not newer, hence the other node keeps them.

1 Like
5

Thanks. I’ve tested like below.

  1. Manually created a file on the Linux server in the synced folder. The file name does not match any ignore pattern.
  2. Wait for the file to be indexed by Syncthing on Linux server.
  3. No same file was created on my laptop, so the override button showed up. I just clicked the override change button.

Now on the Linux server, the file created in #1 was gone. That is exactly as I’ve expected.

This is essentially the same as what happened in my original post - files existed on the Linux server side but not on the laptop, so the laptop think the index is newer on the other side thus provided override button. But the result is different after override.

6

I think the behavior matches my expectations in both cases. Syncthing deleting files that you have told it to ignore would be astonishing to me, and dangerous. Syncthing doesn’t know why you ignored the files - could be because they are crap, or too important to let Syncthing ever touch…

7

OK, so I’ve done some further test and found this.

Just similar to what I’ve tested in my 3rd post. The only difference is this time I manually created a file in the slave folder with a name that matches the ignore pattern on the MASTER NODE. Repeat - the file is created on the slave node, which has an empty ignore list, but it has a name that matches the ignore pattern on the master node.

Now this time the file will not disappear after overridden on master node.

So this leads to conclusions as below.

  1. When the ignore list is empty on the slave node side, it truly will not ignore anything, will pick any file in the folder and index it.
  2. However, the ignore list on the master node do have some effect on the slave node, even though ignore files are not synced across nodes.
  3. For files on slave node that match ignore pattern on master side, the slave will be “aware of a file is ignored on master side, still index it on slave side, and not touch it in slave side if master says override”
  4. For files on slave node that DO NOT match ignore pattern on master side, the slave will “simply delete it if master says override”

#3 actually is quite confusing. I don’t know whether this was designed for a purpose or not. But it makes slave node’s behavior implicitly depends on master’s ignore patterns, which is unexpected as I expect slave node’s behavior should only be related to its own ignore list.

8

I think to summarize, if the node is aware of the files, it will deal with the files, if the node is not aware of the files, it will not deal with the files, and that summarizes the behaviour?

9

Well, the problem is how “awareness state” is maintained. In the case of my OP & Test 2, those files are aware by the slave node during index stage. The appearing of the override change button on master proved it(the slave has some files that the master does not have because master ignored them but slave does not, so master thought slave version is newer thus provided a override option). But later in the “deal with” stage, slave didn’t do anything when master asked to override as if it was not aware of those files.

To summarize, in the index stage, slave node used its own ignore list to determine whether files should be aware or not. But later in the “deal with” stage, slave used master’s ignore list instead of its own.

10

The slave does not know what masters ignore list is, hence there is no way it could have used it. It’s not even part of the protocol. I am still struggling to understand the issue you are referring to.

11

I think not actually. Slave has no idea that other node is master. If you change something on slave, he (slave) will index it (except what’s ignored) and will try to propagate that changes. Master will except that, but he will not touch files. Instead he will offer override button. If you choose to override, he (master) will change version vectors of files that were changed (in the past I would write “increase version number”) and then send info about what he has and what he has not (except what’s ignored) or maybe he sends only info concerning files changed by slave. So to slave it seems that master just changed whole bunch of stuff and deleted some. He (slave) has no means to determine that it was override push, because it’s not part of protocol as Audrius wrote.

Anybody who understands this better than me feel free to correct me.

12

Well this actually is a good news for me as I now know this is not what Syncthing intended to do, it is a bug.

Let me offer a quick test case so everybody who is concerned in this case can try and understand it. Just follow below steps.

  1. Create an empty sync folder pair on node A & B. Check folder master on A. By default the there is no .stignore file so there is no ignore pattern on either A or B.
  2. Create empty file Test1.txt on A. Wait for a while, it will appear on B. This is normal.
  3. Create the .stignore file on A and put “/Test1.txt” in it. This makes A ignore the file created in last step.
  4. Open Test1.txt on A and type in something. Save and close it.
  5. Create empty file Test2.txt on A.
  6. Wait for a while and Test2.txt appears on B. Test1.txt on B is still empty. This is normal as Test1.txt is in A’s ignore list now so A will not transfer any content update of Test1.txt. This proves the ignore pattern on A is correct and working.
  7. Delete Test1.txt from A. Wait for a while.
  8. The Override Change button appears on A. Click the Out of Sync link on A will list Test1.txt. This is normal as B has Test1.txt but A though it don’t have(because ignored).
  9. Click the override button on A. It will disappear for one second but back again in the next second. You can click the override button again and again, it always comes back. The Test1.txt is always listed as out of sync. The Test1.txt file never get deleted on B. This is not normal, and this is the issue I’m talking about.
  10. Create empty file Test3.txt on B.
  11. Now on A the out of sync link shows both Test1.txt and Test3.txt.
  12. Click the override button on A. Test3.txt disappears from out of sync link and B’s folder. This is normal.

#9 is the issue. #12 is the behavior I expect should have happened in #9.

The only difference of Test1.txt and Test3.txt is Test1 is in A(masters)'s ignore list, Test3 is not.

This can be reproduced in 0.10.2x/30 and 0.11.0.

1 Like
13

Thanks, this makes things much more clear. I am not sure what the correct solution is though, I think in step 9, B should not look out of sync to A, and should not suggest to override changes, given it’s a file which is ignored on A.

I’ve opened an issue:

In hopes that @calmh can make a decision on what’s right and what’s wrong.

14

Thanks.

To myself handling #9 as #12 is more practical as it helps to really make slave keep its content in sync with master.

15

Hi @calmh, actually deleting the content of folders that you told syncthing to not keep syncronised is the standard behaviour for other tools. For example in Dropbox it works like this. You can see the whole directory structure and then decide which directories you are going to keep on your computer. The once you are not keeping on your computer will be deleted, but fear not, you can redownload them in any moment. Any other behaviour sounds counterintuitive. Before deleting content it does alert you. But only once per change in what you are going to download.

Which is also why both my and @kraml assumed this was the standard way of acting.

On Syncthing, it could delete something only if a copy of that file is on the other side (which is what I expected it to do). This would make it much more secure. The files listed in ignore would then be something akin of “I don’t want a copy of those in this computer”.

I understand this might be different from how it is working now, but this is the standard way selective sync works. So if you don’t implement it I expect people will ask for it on, and on, and on … ;-).

The problem is that if I need to delete locally the files I decided to ignore, I have to check for every file if it is in the ignore list. And every time I make an error I delete a file that will be then deleted in the other computer. Now this is really dangerous.

16

Sorry for jumping to discussion, but I feel your comparison is not relevant.

Ignore list does not tell ST not to keep synchronised, but not to see them at all.

I feel like Dropbox cannot be compared with ST in this because

  1. Dropbox has a central repository which you cannot control and which have all files and all history of those files.
  2. Dropbox has pretty low size limit (in free version at least) so you sometimes have to choose which subfolders of some big shared folder you want in your pc.

Yes, but with ST other side might not have those files (ignored there too or you are master). I know that you proposed this to solve it:

but I feel it is lot of code in the wrong direction. Main problem I see in this:

Ignore list is not selective sync, probably was never intended to be (I’m guessing from what I read in Selective Sync? - #7 by craibuc and Selective Sync · Issue #193 · syncthing/syncthing · GitHub). There are too many possible problems with it and too much work involved. I think result of discussion was that it would have to be completely rewritten (not based or derived from current implementation of ignore list) and that it is far future. And ST is still 0.X version.

1 Like
17

Yes, I am also realising it. Unfortunately when someone asks if SyncThing has selective sync the answer is to use the ignore list. So the risk is that although it is not selective sync it prevents a real selective sync from ever been written.

1 Like
18

There is no risk of that. On the contrary, there is work ongoing on a more Dropbox-style selective sync. Once that gets in place, if I got to choose I’d rip out and throw away most of the ignore system. It’s too complex and unintuitive for my tastes. Most ignore systems only work in one direction (i.e. rsync, git, …) and then it’s easy to understand what happens. Syncthing is not that…

3 Likes
19

thanks @calmh. That’s very reassuring. Anything we might do to help you and the team on that front, let us know.

1 Like
20

I just want to mention a feature of ownCloud client ignore list, in case it adds to the discussion. Items in the ownCloud client’s ignore list can be marked such that the client is allowed to delete them if they are preventing removal of a directory. This approach would resolve #826.

2 Likes
21

This is probably something we should emulate.

22

Could you have a look at https://github.com/Zillode/syncthing/commit/10021c97a66dba283aa558ca08371a57c69448b8 and tell us what’s missing? I cannot reproduce the behavior you describe and everything works as intended.