Some thoughts on ignores syncing, includes, and defaults.

(Evgeny Kuznetsov) #1

This post is rather prompted by conversation in issue #5615, but my thoughts are really only tangential to the topic of said issue, so they go here instead of there.

There’s obviously some demand for more straitforward ignore patterns syncing. @AudriusButkevicius makes a perfectly valid point in the aforementioned issue: there is a mechanism in place, namely #include in .stignore. However, as far as I’m concerned, this mechanism lacks one crucial feature.

What I suggest is have .stignore autogenerated upon folder setup, and have it have one line by default: #include .stglobalignore.

Now, this is currently not possible, as #include of a non-existing file makes Syncthing throw up, and I’ve seen a combination of SyncTrayzor and Syncthing 1.0.1 bring a Win7 system to its knees with CPU usage under this very condition (unfortunately, I no longer have access to that system to debug and investigate, so no bug report pending here). So the first step would be to make less of a deal out of non-existing file #includes (IMO these deserve a line in the log and proceeding, not halting the folder).

Granted, it makes no sense to even attempt to implement such changes if there’s some actual WIP on issue 2491 that I’m not aware of.

But just in case there isn’t, I would like to go about implementing these two changes:

  1. Make #include foo generate a log warning, not a current folder-stop error in case foo does not exist.
  2. Make .stignore generated upon new folder creation or adding with the default line of #include .stglobalignore.

What good it will do:

  • The default behaviout doesn’t change. As long as no .stglobalignore exists, the autogenerated .stignore is as good as empty, which is the curreng behaviour.
  • If a folder has .stglobalignore, those ignore patterns will automatically propagate to new devices as soon as this file is synced. Perhaps, some files that are supposed to be ignored will get synced before that, but that’s current behaviour anyway, so no regression here.

What bad it will do:

  • People who want something like this will have it and be less interested in getting some work on New Ignore System done, which may further delay New Ignore System.
  • Probably something else I couldn’t think of.

The last sentence above is really why I’m posting this before rushing to implement. What do I miss? Is this a dumb idea generally? Obviously, there’s something bad in it, there should be a reason no one has done it yet, but what is it?

(Simon) #2

I personally would very much dislike this, as I don’t want the folder to start hashing and syncing all content because somehow, accidentally, the included file got deleted.

What I could see is that if that file does not exist, Syncthing checks if it exists on a connected device, and if so pulls it down - that would be handy, but probably quite a bit of effort compared to the usefulness.

A more generally approach would be to implement a modifiable .stignore template. What I mean is have a ignore file in the config folder, who’s contents get copied to .stignore of a newly created folder. A user can then put whatever they want (including a #include .stglobalignore) in there. An alternative approach would be to include those patterns implicitly, but that has to complications: Do they take precedent or not to the folder local .stignore patterns and it’s not immediately evident that they exist when altering patterns in .stignore.

1 Like
(Evgeny Kuznetsov) #3

Point taken.

Not really, no, seeing as how it would stall any new folder by default (since it would not have .stglobalignore in it). For it to work we will either need to pre-fetch all #included files before syncing the folder itself (“quite a bit of effort compared to the usefulness”, as one of the maintainers said), or have those templated as well, which opens a whole separate can of worms when time comes for those #included files to be synced between devices.

Basically, the whole thing boils down to this very issue: global ignore rules have to be synced before they are acted upon, but current ignore system just stops the folder completely before they can be synced if they are initially not there.

Current workaround is really a dirty hack: sync the folder first, manually add global ignore rules (by #including) later. It works, yes, but requires user intervention after initial sync, which can take quite some time and defeats the “set up and forget” model we are supposedly aiming for.

I only see 3 possible approaches to fixing this:

  1. What’s proposed in initial post, i.e. ignore the non-existing #includes. Can result in excessive scanning, syncing, and ultimately leaking data to other devices in case of accidental file deletion (your point, comlpetely valid) - not a good option.
  2. A brand new global ignore system that’s handled separately on protocol level. Requires changing protocol, hence shouldn’t be among the first things to consider.
  3. What you call “an alternative approach”: have a standart file (.stglobalignore or that gets included if exists and doesn’t stall the folder if it doesn’t. This would require explicit documentation and corresponding UI so that it’s obvious enough to the user. It might even be a good idea to make it optional (i.e. a checkbox in UI on both folder editing and folder adding/creating: “use global ignore rules if they exist”) and have some interface to edit it (along with current interface to edit local ignore patterns). This requires arbitrary decisions to be made: do global ignore rules get processed before or after local ignore rules (I suggest after, this way local ignore rules generally take precedence).

If you think the third approach is sane enough, I will start working on it.

As far as I understand in the current ignore pattern system ignore patterns can not really possibly conflict (correct me it I’m wrong), so whatever the content of .stignore and they can be concatenated without new issues emerging (as long as the order is decided upon and documented). In the same way, in case of a sync-conflict on it should be safe to just concatenate all the available versions, shouldn’t it?

1 Like
(Audrius Butkevicius) #4

I see the options but not the rationale why the existing system/workaround is not sufficient.

(Evgeny Kuznetsov) #5

Current workaround:

  1. requires user to put a line in .stignore rather than check a box in UI,
    • Goal #3 of the project: Syncthing should be approachable, understandable and inclusive.
  2. requires user to do it after initial sync is completed (which may be quite some time, days or weeks in some cases) rather than make all the setup on a folder in one go,
    • Goal #4 of the project: User interaction should be required only when absolutely necessary.
1 Like
(Audrius Butkevicius) #6

So I disagree. The user can pre-create both the ignores and ignores global file. I don’t see how clicking some checkbox is less than having to type something in a file.

(Simon) #7

The cool thing for me would be adding a new node to an existing network. Today you need to somehow transfer that global ignore file ahead or retype it or whatever, with this feature it would just be checking a box that you want to honor global ignore patterns.

If I get it correctly you want to designate a file named stignore-global (or similar) as folder specific global ignore patterns, which is synced through regular means. If it exists, apply it, if it doesn’t, just neglect it. Potentially have some control of whether to apply it at all and for editing in the web UI. Right?
If so the behaviour when adding a new device with such global ignores is still not very clean: At the start there is none, so Syncthing starts pulling all files. During that pull it encounters that global ignores file, but it’s just a regular file, so proceed. That could be changed to stop pulling, reread ignore patterns and start again. There is still no guarantee that will happen first. So you could end up with pulling in some files, then immediately ignoring them. So you’d also need something like I mentioned before, that checks whether that file exists on a remote and pulls it in ahead of pulling (and scanning ideally).

(David R) #8

As a new user, this seems like a good idea. The ignore system seems a little too static. I recently noticed a behavior that seems problematic but in the other direction. I created a test directory with two JPEG files and two PNG files. I then installed Syncthing on a second machine and synced that directory. all four files came over. Then, on the second machine, I added *.png to the ignore pattern file. The two PNG files remained, so I deleted them manually. So far so good, but then I removed *.png from the ignore file. Syncthing could then “see” the missing PNG files and propagated that to the other system as a delete, resulting in the PNG files disappearing from the first system. That’s probably not a desired behavior in general.

Perhaps it could benefit from being a little more dynamic in both directions. If something is added to ignore, then all memory of those files ever being synced in the first place should be erased, so that unintended deletes don’t get propagated. Likewise, if something is added to include, then an immediate attempt could be made to sync that file if it exists on the other system, which would result in either a transfer or a conflict right away, but not a surprise later on.

(Vincent Ardern) #9

The user can pre-create both the ignores and ignores global file. I don’t see how clicking some checkbox is less than having to type something in a file.

But what am I typing in that file? It it’s a checkbox I don’t need to keep a copy of my ignore patterns somewhere else so that I can refer to what on earth I’m supposed to be typing/copying into those files. A checkbox would make Syncthing more self contained. As it is, I use Google Drive to remind me what is supposed to go into new Syncthing ignore files. This is a silly dependency to have and ways to improve this would be welcome. If I’m doing it wrong then Syncthing is clearly not as understandable as it could be. I feel a checkbox that I could toggle to grab ignore patterns from elsewhere in the cluster would improve this whole situation by reducing external dependencies, or by making it more understandable. Either outcome is an improvement to my mind.

1 Like
(Evgeny Kuznetsov) #10

Basically, yes. Initial implementation would have it off by default, with a per-folder config option to enable. It can be enabled by default further down the road, if deemed appropriate.

Yes, that could be a sane idea after all. I haven’t yet looked at that part of the code, but seeing as how we have options for file syncing order (alphabetical vs random) it probably can be done.

I can potentially envision corner cases where the behaviour would still be unclean, with stignore-global itself being ignored by local ignore patterns or accidentaly deleted. Also, a decision should be made how to handle conflicts on stignore-global.