Some thoughts on ignores syncing, includes, and defaults.

Point taken.

Not really, no, seeing as how it would stall any new folder by default (since it would not have .stglobalignore in it). For it to work we will either need to pre-fetch all #included files before syncing the folder itself (“quite a bit of effort compared to the usefulness”, as one of the maintainers said), or have those templated as well, which opens a whole separate can of worms when time comes for those #included files to be synced between devices.

Basically, the whole thing boils down to this very issue: global ignore rules have to be synced before they are acted upon, but current ignore system just stops the folder completely before they can be synced if they are initially not there.

Current workaround is really a dirty hack: sync the folder first, manually add global ignore rules (by #including) later. It works, yes, but requires user intervention after initial sync, which can take quite some time and defeats the “set up and forget” model we are supposedly aiming for.

I only see 3 possible approaches to fixing this:

  1. What’s proposed in initial post, i.e. ignore the non-existing #includes. Can result in excessive scanning, syncing, and ultimately leaking data to other devices in case of accidental file deletion (your point, comlpetely valid) - not a good option.
  2. A brand new global ignore system that’s handled separately on protocol level. Requires changing protocol, hence shouldn’t be among the first things to consider.
  3. What you call “an alternative approach”: have a standart file (.stglobalignore or .stignore.global) that gets included if exists and doesn’t stall the folder if it doesn’t. This would require explicit documentation and corresponding UI so that it’s obvious enough to the user. It might even be a good idea to make it optional (i.e. a checkbox in UI on both folder editing and folder adding/creating: “use global ignore rules if they exist”) and have some interface to edit it (along with current interface to edit local ignore patterns). This requires arbitrary decisions to be made: do global ignore rules get processed before or after local ignore rules (I suggest after, this way local ignore rules generally take precedence).

If you think the third approach is sane enough, I will start working on it.

As far as I understand in the current ignore pattern system ignore patterns can not really possibly conflict (correct me it I’m wrong), so whatever the content of .stignore and .stignore.global they can be concatenated without new issues emerging (as long as the order is decided upon and documented). In the same way, in case of a sync-conflict on .stignore.global it should be safe to just concatenate all the available versions, shouldn’t it?

1 Like

I see the options but not the rationale why the existing system/workaround is not sufficient.

Current workaround:

  1. requires user to put a line in .stignore rather than check a box in UI,
    • Goal #3 of the project: Syncthing should be approachable, understandable and inclusive.
  2. requires user to do it after initial sync is completed (which may be quite some time, days or weeks in some cases) rather than make all the setup on a folder in one go,
    • Goal #4 of the project: User interaction should be required only when absolutely necessary.
1 Like

So I disagree. The user can pre-create both the ignores and ignores global file. I don’t see how clicking some checkbox is less than having to type something in a file.

The cool thing for me would be adding a new node to an existing network. Today you need to somehow transfer that global ignore file ahead or retype it or whatever, with this feature it would just be checking a box that you want to honor global ignore patterns.

@nekr0z
If I get it correctly you want to designate a file named stignore-global (or similar) as folder specific global ignore patterns, which is synced through regular means. If it exists, apply it, if it doesn’t, just neglect it. Potentially have some control of whether to apply it at all and for editing in the web UI. Right?
If so the behaviour when adding a new device with such global ignores is still not very clean: At the start there is none, so Syncthing starts pulling all files. During that pull it encounters that global ignores file, but it’s just a regular file, so proceed. That could be changed to stop pulling, reread ignore patterns and start again. There is still no guarantee that will happen first. So you could end up with pulling in some files, then immediately ignoring them. So you’d also need something like I mentioned before, that checks whether that file exists on a remote and pulls it in ahead of pulling (and scanning ideally).

As a new user, this seems like a good idea. The ignore system seems a little too static. I recently noticed a behavior that seems problematic but in the other direction. I created a test directory with two JPEG files and two PNG files. I then installed Syncthing on a second machine and synced that directory. all four files came over. Then, on the second machine, I added *.png to the ignore pattern file. The two PNG files remained, so I deleted them manually. So far so good, but then I removed *.png from the ignore file. Syncthing could then “see” the missing PNG files and propagated that to the other system as a delete, resulting in the PNG files disappearing from the first system. That’s probably not a desired behavior in general.

Perhaps it could benefit from being a little more dynamic in both directions. If something is added to ignore, then all memory of those files ever being synced in the first place should be erased, so that unintended deletes don’t get propagated. Likewise, if something is added to include, then an immediate attempt could be made to sync that file if it exists on the other system, which would result in either a transfer or a conflict right away, but not a surprise later on.

1 Like

The user can pre-create both the ignores and ignores global file. I don’t see how clicking some checkbox is less than having to type something in a file.

But what am I typing in that file? It it’s a checkbox I don’t need to keep a copy of my ignore patterns somewhere else so that I can refer to what on earth I’m supposed to be typing/copying into those files. A checkbox would make Syncthing more self contained. As it is, I use Google Drive to remind me what is supposed to go into new Syncthing ignore files. This is a silly dependency to have and ways to improve this would be welcome. If I’m doing it wrong then Syncthing is clearly not as understandable as it could be. I feel a checkbox that I could toggle to grab ignore patterns from elsewhere in the cluster would improve this whole situation by reducing external dependencies, or by making it more understandable. Either outcome is an improvement to my mind.

1 Like

Basically, yes. Initial implementation would have it off by default, with a per-folder config option to enable. It can be enabled by default further down the road, if deemed appropriate.

Yes, that could be a sane idea after all. I haven’t yet looked at that part of the code, but seeing as how we have options for file syncing order (alphabetical vs random) it probably can be done.

I can potentially envision corner cases where the behaviour would still be unclean, with stignore-global itself being ignored by local ignore patterns or accidentaly deleted. Also, a decision should be made how to handle conflicts on stignore-global.

I really like the idea of having a syncing sync-ignore-list.

This is what I fiddled together to have a “global” and a “specific” ignore list.

.stignore:

!.stignore_global
!.stignore_specific

#include .stignore_global
#include .stignore_specific

.stignore_global:

(?d)(?i).DS_Store
(?d)(?i).Spotlight-V100
(?d)(?i).Trashes
(?d)(?i)desktop.ini
(?d)(?i)Thumbs.db

.stignore_specific:

/Test
/iCloud Photos

This concept works out for me but it took me quite a while to arrange.

I support the idea of inviting & helping a user to sync their ignore lists innately, because non-technical users have a hard time understanding and creating those. I know, because I’m one of them!

With additional nice comments and some examples in the .stglobalignore file, the user could get the idea behind it even quicker/better than reading the forum and the docs only.

Four years in a row now I’ve gone to my parents’ place for Christmas and fixed their computers; 4 years I’ve ended up reinstalling the OS for some reason or another, which results in reinstalling syncthing - and 4 years now that I’ve shot myself in the foot by forgettting to go into the advanced settings to type #include .stglobalignore each time I sync a new folder…

I see there’s a bountysource bounty for the “Next Gen Ignores” task which has a bunch of other stuff I don’t care about, and this one feature that I DO care about is a “maybe” :frowning: If there’s some way to me to donate towards this specific feature (“Act as if .stignore contained #include .stglobalignore by default”), I would love to do so <3

Initial implementation would have it off by default, with a per-folder config option to enable

I’m not sure why anyone would want it off by default (If you don’t want your repo to have global ignores, then don’t create a .stglobalignore file?) - but if this opt-in approach DOES get taken, it’d be nice to detect the presence of a .stglobalignore file and ask the user if they would like to use it - otherwise we’re just replacing “I forgot to go into the advanced settings and type #include .stglobalignore into the ignore patterns box” with “I forgot to go into the advanced settings and tick the Use .stglobalignore checkbox”…

1 Like

I guess you symlink this file as it appears to be in the folder root? And don’t have any other things in the ignore file as this is the only thing you need to remember to type in? So you could just symlink .stignore instead?

What about a default ignore file for new folders? Can exist in the config dir and also be edited in the UI. You can still forget to create that, but at least you only need to do it once.

1 Like

Sounds workable, though I prefer a config field over a new file.

I kinda expected dislike for a file based solution :slight_smile:

The advantage for me is i can just copy and edit a plain text file - simpler than interacting with a UI or xml. Anyway it doesn’t actually matter.

Yeah and it has its own format (stignores), which is nasty to embed in another format (xml). <ignores><![CDATA[asdf\n!gronk]]></ignores> vs <ignores><pattern>asdf</pattern><pattern exclude="true">gronk</pattern></ignores>? :slight_smile:

I’m not sure what would be symlinked to what here? I could symlink .stignore to .stglobalignore instead of using #include, but I’m not sure how that relates to the issue of “each time I set up a new PC, I need to remember to go and twiddle some extra settings somewhere else to enable my standard ignore list”

And don’t have any other things in the ignore file as this is the only thing you need to remember to type in?

That is correct - I’ve not yet found a use-case for local ignores, but pretty much all of my folders have global ignores in them (Code folder ignores .cache, Downloads folder ignores *.partial, Photos folder ignores Thumbs.db and .DS_Store, etc)

#include'ing a synced file is the best way I’ve found of making that happen - The only pain-point with this approach is that I consistently forget to enable it :frowning:

(With the consequence that junk which is supposed to be ignored gets into my metadata index - so I go back and add the ignores, delete the files… and then the ghosts of these files haunt me because of file names lingering forever and my repos are marked as “out of sync” until I nuke the whole setup and configure from scratch with a new folder ID)

Your example is #include .stglobalignore. That presumes the existence of the .stglobalignore file, which is in the folder root. Either you then copy this into each folder from somewhere, or you create a symlink pointing to it. Regardless of which it is, it seems you could do this directly for .stignore instead, since you don’t use any folder-local ignore patterns.

Either you then copy this into each folder from somewhere, or you create a symlink pointing to it

Eh? .stglobalignore is part of the folder being sync’ed, so syncthing copies it from my fully-configured NAS to my freshly-installed desktop

it seems you could do this directly for .stignore instead

I wish I could! That would make my use-case so much simpler, but: https://github.com/syncthing/syncthing/blob/486230768e5bea25fc758c119f9b9a9c6d529af3/lib/fs/filesystem.go#L207

Ah, ok, that makes sense. However it also results in the usual chicken and egg problem of not having the ignores available until after the folder has synced. I guess this is fine in your case if the ignores are always symmetrical. Regardless, it sounds like imsodins suggestion would be helpful.

1 Like

Just set up a new laptop; synced my files from my NAS; made sure everything was sync’ed correctly, all looks good. A week later, I notice there’s some weirdness where files that are in my ignore list aren’t being ignored… guess what I forgot to enable AGAIN? T_T