First of all I would like to thank you for this great software. This is really a wonderful tool.
I have run into a specific problem in my setup which I am not sure whether is a bug or just as expected. I can’t find a similar topic in the forum so would like to ask here.
I use Syncthing to sync drives on my Windows laptop to a Linux file server. I’ve started using since 0.10.x and now on the latest 0.11. The problem is always there. My situation is,
I created a sync folder entry on my laptop. – Edit: and it is set as mater folder.
I started to sync it with my Linux server.
Everything is fine so far.
I noticed some files are just temporary so decided to add them in the .stignore file.
Those ignored files are no longer synced because newly created ones that matches the pattern I’ve added not appear on the other side.
However those temporary files that were synced before I put the pattern in, are already on the Linux server and are not deleted automatically.
The override change button on my laptop now appears. So I clicked it.
The result I expected - Those files I mentioned in #6 should be deleted on the Linux server because they are “overridden”.
Actual result - Nothing happened. These files are still there. And the override change button is still on. I clicked it again and again, it is always the same.
I would consider step 4 a rather common operation. I have many files on my laptop and I will not review them all before I start to sync. I just start to sync with a empty .stignore file and later begin to put in patterns as I notice some file is not worth syncing. With this approach I end up with a lot useless files on the other side and have no easy way to clean them up. If this is a bug I would like it be fixed. And if it is how Syncthing works currently, I hope it could be enhanced to handle such situation.
Ignore patterns simply hides the files from syncthing, so as far as syncthing is concerned, the files don’t exist on either side (though they exist on the filesystem and you have to delete them yourself).
Thanks for your reply. I understand what you mentioned and here is my thinking.
I’ve only added the ignore pattern on my laptop. So for the Syncthing on my laptop, those files matches the pattern “does not exist”. On the Linux server, the .stignore file is empty. So the previously synced files are visible to the Synching on the Linux servers, and in its database already as they are received earlier. And I believe that’s why the override changes button appears on my laptop, because it sees the other node has some files it doesn’t have. As a master it is now able to “override”, to ask the other node to delete those files and keep in line with what the mater have.
Please correct me if my understanding is wrong.
BTW, I forgot to mention that I’ve setup the laptop as mater. I’ll edit it.
Overriding does not delete anything, it just forces everyone to think that the files you have are newer.
Since you technically don’t have the ignored files, they are not newer, hence the other node keeps them.
Manually created a file on the Linux server in the synced folder. The file name does not match any ignore pattern.
Wait for the file to be indexed by Syncthing on Linux server.
No same file was created on my laptop, so the override button showed up. I just clicked the override change button.
Now on the Linux server, the file created in #1 was gone. That is exactly as I’ve expected.
This is essentially the same as what happened in my original post - files existed on the Linux server side but not on the laptop, so the laptop think the index is newer on the other side thus provided override button. But the result is different after override.
I think the behavior matches my expectations in both cases. Syncthing deleting files that you have told it to ignore would be astonishing to me, and dangerous. Syncthing doesn’t know why you ignored the files - could be because they are crap, or too important to let Syncthing ever touch…
OK, so I’ve done some further test and found this.
Just similar to what I’ve tested in my 3rd post. The only difference is this time I manually created a file in the slave folder with a name that matches the ignore pattern on the MASTER NODE. Repeat - the file is created on the slave node, which has an empty ignore list, but it has a name that matches the ignore pattern on the master node.
Now this time the file will not disappear after overridden on master node.
So this leads to conclusions as below.
When the ignore list is empty on the slave node side, it truly will not ignore anything, will pick any file in the folder and index it.
However, the ignore list on the master node do have some effect on the slave node, even though ignore files are not synced across nodes.
For files on slave node that match ignore pattern on master side, the slave will be “aware of a file is ignored on master side, still index it on slave side, and not touch it in slave side if master says override”
For files on slave node that DO NOT match ignore pattern on master side, the slave will “simply delete it if master says override”
#3 actually is quite confusing. I don’t know whether this was designed for a purpose or not. But it makes slave node’s behavior implicitly depends on master’s ignore patterns, which is unexpected as I expect slave node’s behavior should only be related to its own ignore list.
I think to summarize, if the node is aware of the files, it will deal with the files, if the node is not aware of the files, it will not deal with the files, and that summarizes the behaviour?
Well, the problem is how “awareness state” is maintained. In the case of my OP & Test 2, those files are aware by the slave node during index stage. The appearing of the override change button on master proved it(the slave has some files that the master does not have because master ignored them but slave does not, so master thought slave version is newer thus provided a override option). But later in the “deal with” stage, slave didn’t do anything when master asked to override as if it was not aware of those files.
To summarize, in the index stage, slave node used its own ignore list to determine whether files should be aware or not. But later in the “deal with” stage, slave used master’s ignore list instead of its own.
The slave does not know what masters ignore list is, hence there is no way it could have used it. It’s not even part of the protocol.
I am still struggling to understand the issue you are referring to.
I think not actually. Slave has no idea that other node is master. If you change something on slave, he (slave) will index it (except what’s ignored) and will try to propagate that changes. Master will except that, but he will not touch files. Instead he will offer override button. If you choose to override, he (master) will change version vectors of files that were changed (in the past I would write “increase version number”) and then send info about what he has and what he has not (except what’s ignored) or maybe he sends only info concerning files changed by slave. So to slave it seems that master just changed whole bunch of stuff and deleted some. He (slave) has no means to determine that it was override push, because it’s not part of protocol as Audrius wrote.
Anybody who understands this better than me feel free to correct me.
Well this actually is a good news for me as I now know this is not what Syncthing intended to do, it is a bug.
Let me offer a quick test case so everybody who is concerned in this case can try and understand it. Just follow below steps.
Create an empty sync folder pair on node A & B. Check folder master on A. By default the there is no .stignore file so there is no ignore pattern on either A or B.
Create empty file Test1.txt on A. Wait for a while, it will appear on B. This is normal.
Create the .stignore file on A and put “/Test1.txt” in it. This makes A ignore the file created in last step.
Open Test1.txt on A and type in something. Save and close it.
Create empty file Test2.txt on A.
Wait for a while and Test2.txt appears on B. Test1.txt on B is still empty. This is normal as Test1.txt is in A’s ignore list now so A will not transfer any content update of Test1.txt. This proves the ignore pattern on A is correct and working.
Delete Test1.txt from A. Wait for a while.
The Override Change button appears on A. Click the Out of Sync link on A will list Test1.txt. This is normal as B has Test1.txt but A though it don’t have(because ignored).
Click the override button on A. It will disappear for one second but back again in the next second. You can click the override button again and again, it always comes back. The Test1.txt is always listed as out of sync. The Test1.txt file never get deleted on B. This is not normal, and this is the issue I’m talking about.
Create empty file Test3.txt on B.
Now on A the out of sync link shows both Test1.txt and Test3.txt.
Click the override button on A. Test3.txt disappears from out of sync link and B’s folder. This is normal.
#9 is the issue. #12 is the behavior I expect should have happened in #9.
The only difference of Test1.txt and Test3.txt is Test1 is in A(masters)'s ignore list, Test3 is not.
Thanks, this makes things much more clear.
I am not sure what the correct solution is though, I think in step 9, B should not look out of sync to A, and should not suggest to override changes, given it’s a file which is ignored on A.
I’ve opened an issue:
In hopes that @calmh can make a decision on what’s right and what’s wrong.
Hi @calmh, actually deleting the content of folders that you told syncthing to not keep syncronised is the standard behaviour for other tools. For example in Dropbox it works like this. You can see the whole directory structure and then decide which directories you are going to keep on your computer. The once you are not keeping on your computer will be deleted, but fear not, you can redownload them in any moment. Any other behaviour sounds counterintuitive. Before deleting content it does alert you. But only once per change in what you are going to download.
Which is also why both my and @kraml assumed this was the standard way of acting.
On Syncthing, it could delete something only if a copy of that file is on the other side (which is what I expected it to do). This would make it much more secure. The files listed in ignore would then be something akin of “I don’t want a copy of those in this computer”.
I understand this might be different from how it is working now, but this is the standard way selective sync works. So if you don’t implement it I expect people will ask for it on, and on, and on … ;-).
The problem is that if I need to delete locally the files I decided to ignore, I have to check for every file if it is in the ignore list. And every time I make an error I delete a file that will be then deleted in the other computer. Now this is really dangerous.
Sorry for jumping to discussion, but I feel your comparison is not relevant.
Ignore list does not tell ST not to keep synchronised, but not to see them at all.
I feel like Dropbox cannot be compared with ST in this because
Dropbox has a central repository which you cannot control and which have all files and all history of those files.
Dropbox has pretty low size limit (in free version at least) so you sometimes have to choose which subfolders of some big shared folder you want in your pc.
Yes, but with ST other side might not have those files (ignored there too or you are master). I know that you proposed this to solve it:
but I feel it is lot of code in the wrong direction. Main problem I see in this:
Ignore list is not selective sync, probably was never intended to be (I’m guessing from what I read in Selective Sync? and https://github.com/syncthing/syncthing/issues/193). There are too many possible problems with it and too much work involved. I think result of discussion was that it would have to be completely rewritten (not based or derived from current implementation of ignore list) and that it is far future. And ST is still 0.X version.
Yes, I am also realising it. Unfortunately when someone asks if SyncThing has selective sync the answer is to use the ignore list. So the risk is that although it is not selective sync it prevents a real selective sync from ever been written.
There is no risk of that. On the contrary, there is work ongoing on a more Dropbox-style selective sync. Once that gets in place, if I got to choose I’d rip out and throw away most of the ignore system. It’s too complex and unintuitive for my tastes. Most ignore systems only work in one direction (i.e. rsync, git, …) and then it’s easy to understand what happens. Syncthing is not that…
I just want to mention a feature of ownCloud client ignore list, in case it adds to the discussion. Items in the ownCloud client’s ignore list can be marked such that the client is allowed to delete them if they are preventing removal of a directory. This approach would resolve #826.