Syncing ignore patterns

Lately there has been some renewed interest expressed for ignore patterns that sync between devices (https://github.com/syncthing/syncthing/issues/2491). As that’s not necessarily dependent on the huge “Next gen ignores” issue, lets discuss needs and potential ways to implement it. I am interested in this feature (as in I have lots of #include .stignore-shared in my patterns), so if there’s a sensible way forward I may implement it.

Some initial thoughts:

  • Declare another special file like .stignore-shared or ...-global that is treated like a normal file with regards to syncing but is automatically included if present.

  • How to handle conflicts? Maybe make whatever parses ignore patterns look for conflicts and stop the folder in case there is one.

  • How to handle the initial chicken-egg problem of initial scan and then sync needs to happen for the file to be present, but we don’t want to scan/sync some stuff that is ignored? We could look for that file at folder creation time - probably either somewhat hackish or requiring huge refactor.

1 Like

That doesn’t really help if you need to bootstrap that file from a remote machine.

You’d somehow need to get the index entry for that out of bound. Also, you potentially need to get it from everyone, as picking someone specific might not give you the latest version.

It might not be a huge problem if you start empty and get the file from your first peer. Presumably they will have the patterns in effect and files they announce will all be such as are allowed by the patterns.

If you start out with a bunch of files where some of them will be matched by the ignore patterns after syncing … :man_shrugging: yeah

I don’t think we should do anything more than we always do, at least not initially.

One other thing though. It’s not enough to just declare a new filename, we also need to consider how to say that I don’t want to apply the ignore patterns, yet the file is present because someone else is syncing it.

1 Like

Hmm, yeah, that might be a hard one. Either we loose usability by requiring the user to enable this feature, or if the default is enabled that might indeed be annoying in multi-user setups. For default disabled I don’t see any benefit of an additional config over default ignore patterns (which can have #include .stignore-global).

we also need to consider how to say that I don’t want to apply the ignore patterns

I’m curious about the real-world use-case for that, because I don’t understand the problem well enough to suggest a solution -

Assuming that the people I’m sharing with are non-malicious, I’d expect my teammates to respect the “things which everybody should ignore (.cache/*, etc) go in .stignore-global; things which only I want to ignore go in .stignore” system.

In the case of my non-malicious but non-technical family, they aren’t technical enough to use either of those files, so I’m not worried about accidental modification there either.

If I’m sharing with someone malicious, I’d worry about them corrupting my files a lot more than I’d worry about them messing up my ignore list…

loose usability by requiring the user to enable this feature

Personally I’d prefer enabled-by-default (the whole reason I want this feature is because I consistently forget to do non-default things) - but if the decision is to go disabled-by-default, one could perhaps default to null, then if a .stignore-global file is detected, the GUI will pop up a message “It looks like this folder contains a .stignore-global file, do you want to enable or disable that?” and set the value to true/false at that point?

conflicts

for all the use-cases I can imagine, these files get modified very rarely, so conflicts should be very-rare-squared, so dealing with them the same way any other conflict is dealt with should be totally fine; is there a use-case where conflicts might happen often, and cause problems?

Maybe there is no use case, especially as long as the regular local .stignore has priority and can override as required.

1 Like

The bootstrapping problem remains tho. We’d have to fiddle with the index senders presumably, and also go for a pull-when-receiving-index state machine deviation… Or I guess always check this new magical file on pull.

For an initial revision at least I’d say screw that, if you rely on synced ignore patterns you better make sure to not have a fully populated .cache/* or whatever when you start syncing the folder…

It’s also not really a disaster. The files will get scanned, ignored on the other side when sent, then invalidated when the ignore patterns kick in.

2 Likes

3 posts were split to a new topic: Should “meta”-files use .syncthing instead of .st as prefix?

I would love to see this, for example, I made remote backups of ‘devel’ folders, that contain python caches, virtualenvs, etc… it would be great to be able to mark exclusions to be global and have them synced with other R/W hosts so that no one attempts to copy over those files

It should be possible to just use two fields in the GUI for Local Ignore (the current one) and a Global Ignore (the new second). This would then be followed by two files, one of which is copied between the connected devices. That would certainly be a representable standard. Multiple #include entries would then be used for a multi-global ignore environment.

Since more and more system files are added, maybe again we take up the systemic Syncthing subdirectory again.

Hi All!

In my case #include-s work fine except the first sync when I have to exclude an included file from ignores to download it first. Only after that I add it as include and remove ignores.

So, I believe it would be fine if included files will be pulled first regardless of further ignore patterns.

Just yesterday I had an incident.

One node accidentally deleted the file included into ignore patterns. This action was successfully propagated on all other online nodes. So all of them failed into parsing error and syncing stopped.

From the one hand it is correct behavior as I do not want to sync ignored files (the list of which is unknown) with other nodes.

But, form the other hand I can not just restore included file in one node and automatically sync it with another one to return my net into syncing state. I had to restore this file manually on each affected node.

That is why I believe that a separate high priority syncing list for included files is extremely necessary.