This post is rather prompted by conversation in issue #5615, but my thoughts are really only tangential to the topic of said issue, so they go here instead of there.
There’s obviously some demand for more straitforward ignore patterns syncing. @AudriusButkevicius makes a perfectly valid point in the aforementioned issue: there is a mechanism in place, namely #include in .stignore. However, as far as I’m concerned, this mechanism lacks one crucial feature.
What I suggest is have .stignore autogenerated upon folder setup, and have it have one line by default: #include .stglobalignore.
Now, this is currently not possible, as #include of a non-existing file makes Syncthing throw up, and I’ve seen a combination of SyncTrayzor and Syncthing 1.0.1 bring a Win7 system to its knees with CPU usage under this very condition (unfortunately, I no longer have access to that system to debug and investigate, so no bug report pending here). So the first step would be to make less of a deal out of non-existing file #includes (IMO these deserve a line in the log and proceeding, not halting the folder).
Granted, it makes no sense to even attempt to implement such changes if there’s some actual WIP on issue 2491 that I’m not aware of.
But just in case there isn’t, I would like to go about implementing these two changes:
Make #include foo generate a log warning, not a current folder-stop error in case foo does not exist.
Make .stignore generated upon new folder creation or adding with the default line of #include .stglobalignore.
What good it will do:
The default behaviout doesn’t change. As long as no .stglobalignore exists, the autogenerated .stignore is as good as empty, which is the curreng behaviour.
If a folder has .stglobalignore, those ignore patterns will automatically propagate to new devices as soon as this file is synced. Perhaps, some files that are supposed to be ignored will get synced before that, but that’s current behaviour anyway, so no regression here.
What bad it will do:
People who want something like this will have it and be less interested in getting some work on New Ignore System done, which may further delay New Ignore System.
Probably something else I couldn’t think of.
The last sentence above is really why I’m posting this before rushing to implement. What do I miss? Is this a dumb idea generally? Obviously, there’s something bad in it, there should be a reason no one has done it yet, but what is it?
I personally would very much dislike this, as I don’t want the folder to start hashing and syncing all content because somehow, accidentally, the included file got deleted.
What I could see is that if that file does not exist, Syncthing checks if it exists on a connected device, and if so pulls it down - that would be handy, but probably quite a bit of effort compared to the usefulness.
A more generally approach would be to implement a modifiable .stignore template. What I mean is have a ignore file in the config folder, who’s contents get copied to .stignore of a newly created folder. A user can then put whatever they want (including a #include .stglobalignore) in there. An alternative approach would be to include those patterns implicitly, but that has to complications: Do they take precedent or not to the folder local .stignore patterns and it’s not immediately evident that they exist when altering patterns in .stignore.
Not really, no, seeing as how it would stall any new folder by default (since it would not have .stglobalignore in it). For it to work we will either need to pre-fetch all #included files before syncing the folder itself (“quite a bit of effort compared to the usefulness”, as one of the maintainers said), or have those templated as well, which opens a whole separate can of worms when time comes for those #included files to be synced between devices.
Basically, the whole thing boils down to this very issue: global ignore rules have to be synced before they are acted upon, but current ignore system just stops the folder completely before they can be synced if they are initially not there.
Current workaround is really a dirty hack: sync the folder first, manually add global ignore rules (by #including) later. It works, yes, but requires user intervention after initial sync, which can take quite some time and defeats the “set up and forget” model we are supposedly aiming for.
I only see 3 possible approaches to fixing this:
What’s proposed in initial post, i.e. ignore the non-existing #includes. Can result in excessive scanning, syncing, and ultimately leaking data to other devices in case of accidental file deletion (your point, comlpetely valid) - not a good option.
A brand new global ignore system that’s handled separately on protocol level. Requires changing protocol, hence shouldn’t be among the first things to consider.
What you call “an alternative approach”: have a standart file (.stglobalignore or .stignore.global) that gets included if exists and doesn’t stall the folder if it doesn’t. This would require explicit documentation and corresponding UI so that it’s obvious enough to the user. It might even be a good idea to make it optional (i.e. a checkbox in UI on both folder editing and folder adding/creating: “use global ignore rules if they exist”) and have some interface to edit it (along with current interface to edit local ignore patterns). This requires arbitrary decisions to be made: do global ignore rules get processed before or after local ignore rules (I suggest after, this way local ignore rules generally take precedence).
If you think the third approach is sane enough, I will start working on it.
As far as I understand in the current ignore pattern system ignore patterns can not really possibly conflict (correct me it I’m wrong), so whatever the content of .stignore and .stignore.global they can be concatenated without new issues emerging (as long as the order is decided upon and documented). In the same way, in case of a sync-conflict on .stignore.global it should be safe to just concatenate all the available versions, shouldn’t it?
requires user to put a line in .stignore rather than check a box in UI,
Goal #3 of the project: Syncthing should be approachable, understandable and inclusive.
requires user to do it after initial sync is completed (which may be quite some time, days or weeks in some cases) rather than make all the setup on a folder in one go,
Goal #4 of the project: User interaction should be required only when absolutely necessary.
So I disagree. The user can pre-create both the ignores and ignores global file. I don’t see how clicking some checkbox is less than having to type something in a file.
The cool thing for me would be adding a new node to an existing network. Today you need to somehow transfer that global ignore file ahead or retype it or whatever, with this feature it would just be checking a box that you want to honor global ignore patterns.
@nekr0z
If I get it correctly you want to designate a file named stignore-global (or similar) as folder specific global ignore patterns, which is synced through regular means. If it exists, apply it, if it doesn’t, just neglect it. Potentially have some control of whether to apply it at all and for editing in the web UI. Right?
If so the behaviour when adding a new device with such global ignores is still not very clean: At the start there is none, so Syncthing starts pulling all files. During that pull it encounters that global ignores file, but it’s just a regular file, so proceed. That could be changed to stop pulling, reread ignore patterns and start again. There is still no guarantee that will happen first. So you could end up with pulling in some files, then immediately ignoring them. So you’d also need something like I mentioned before, that checks whether that file exists on a remote and pulls it in ahead of pulling (and scanning ideally).
As a new user, this seems like a good idea. The ignore system seems a little too static. I recently noticed a behavior that seems problematic but in the other direction. I created a test directory with two JPEG files and two PNG files. I then installed Syncthing on a second machine and synced that directory. all four files came over. Then, on the second machine, I added *.png to the ignore pattern file. The two PNG files remained, so I deleted them manually. So far so good, but then I removed *.png from the ignore file. Syncthing could then “see” the missing PNG files and propagated that to the other system as a delete, resulting in the PNG files disappearing from the first system. That’s probably not a desired behavior in general.
Perhaps it could benefit from being a little more dynamic in both directions. If something is added to ignore, then all memory of those files ever being synced in the first place should be erased, so that unintended deletes don’t get propagated. Likewise, if something is added to include, then an immediate attempt could be made to sync that file if it exists on the other system, which would result in either a transfer or a conflict right away, but not a surprise later on.
The user can pre-create both the ignores and ignores global file. I don’t see how clicking some checkbox is less than having to type something in a file.
But what am I typing in that file? It it’s a checkbox I don’t need to keep a copy of my ignore patterns somewhere else so that I can refer to what on earth I’m supposed to be typing/copying into those files. A checkbox would make Syncthing more self contained. As it is, I use Google Drive to remind me what is supposed to go into new Syncthing ignore files. This is a silly dependency to have and ways to improve this would be welcome. If I’m doing it wrong then Syncthing is clearly not as understandable as it could be. I feel a checkbox that I could toggle to grab ignore patterns from elsewhere in the cluster would improve this whole situation by reducing external dependencies, or by making it more understandable. Either outcome is an improvement to my mind.
Basically, yes. Initial implementation would have it off by default, with a per-folder config option to enable. It can be enabled by default further down the road, if deemed appropriate.
Yes, that could be a sane idea after all. I haven’t yet looked at that part of the code, but seeing as how we have options for file syncing order (alphabetical vs random) it probably can be done.
I can potentially envision corner cases where the behaviour would still be unclean, with stignore-global itself being ignored by local ignore patterns or accidentaly deleted. Also, a decision should be made how to handle conflicts on stignore-global.
This concept works out for me but it took me quite a while to arrange.
I support the idea of inviting & helping a user to sync their ignore lists innately, because non-technical users have a hard time understanding and creating those. I know, because I’m one of them!
With additional nice comments and some examples in the .stglobalignore file, the user could get the idea behind it even quicker/better than reading the forum and the docs only.
Four years in a row now I’ve gone to my parents’ place for Christmas and fixed their computers; 4 years I’ve ended up reinstalling the OS for some reason or another, which results in reinstalling syncthing - and 4 years now that I’ve shot myself in the foot by forgettting to go into the advanced settings to type #include .stglobalignore each time I sync a new folder…
I see there’s a bountysource bounty for the “Next Gen Ignores” task which has a bunch of other stuff I don’t care about, and this one feature that I DO care about is a “maybe” If there’s some way to me to donate towards this specific feature (“Act as if .stignore contained #include .stglobalignore by default”), I would love to do so <3
Initial implementation would have it off by default, with a per-folder config option to enable
I’m not sure why anyone would want it off by default (If you don’t want your repo to have global ignores, then don’t create a .stglobalignore file?) - but if this opt-in approach DOES get taken, it’d be nice to detect the presence of a .stglobalignore file and ask the user if they would like to use it - otherwise we’re just replacing “I forgot to go into the advanced settings and type #include .stglobalignore into the ignore patterns box” with “I forgot to go into the advanced settings and tick the Use .stglobalignore checkbox”…
I guess you symlink this file as it appears to be in the folder root? And don’t have any other things in the ignore file as this is the only thing you need to remember to type in? So you could just symlink .stignore instead?
What about a default ignore file for new folders? Can exist in the config dir and also be edited in the UI. You can still forget to create that, but at least you only need to do it once.
Yeah and it has its own format (stignores), which is nasty to embed in another format (xml). <ignores><![CDATA[asdf\n!gronk]]></ignores> vs <ignores><pattern>asdf</pattern><pattern exclude="true">gronk</pattern></ignores>?
I’m not sure what would be symlinked to what here? I could symlink .stignore to .stglobalignore instead of using #include, but I’m not sure how that relates to the issue of “each time I set up a new PC, I need to remember to go and twiddle some extra settings somewhere else to enable my standard ignore list”
And don’t have any other things in the ignore file as this is the only thing you need to remember to type in?
That is correct - I’ve not yet found a use-case for local ignores, but pretty much all of my folders have global ignores in them (Code folder ignores .cache, Downloads folder ignores *.partial, Photos folder ignores Thumbs.db and .DS_Store, etc)
#include'ing a synced file is the best way I’ve found of making that happen - The only pain-point with this approach is that I consistently forget to enable it
(With the consequence that junk which is supposed to be ignored gets into my metadata index - so I go back and add the ignores, delete the files… and then the ghosts of these files haunt me because of file names lingering forever - #5 by shish and my repos are marked as “out of sync” until I nuke the whole setup and configure from scratch with a new folder ID)
Your example is #include .stglobalignore. That presumes the existence of the .stglobalignore file, which is in the folder root. Either you then copy this into each folder from somewhere, or you create a symlink pointing to it. Regardless of which it is, it seems you could do this directly for .stignore instead, since you don’t use any folder-local ignore patterns.
If there’s some way to me to donate towards this specific feature (“Act as if .stignore contained 