Proposal: Better management of ignore patterns

Feature
1

Copied from this post on GitHub.

Original Title: Proposal: Support absolute include paths in .stignore

Background

I was setting up Syncthing again this week (old laptop died). I created ~/.stignoreglobal on my new Windows machine and went to work adding all my existing folders. I was quickly reminded that there is no way to #ignore a file in each folder’s .stignore using an absolute path. While not the end of the world it does make setting up the .stignore in folders with various degrees of nesting a huge pain. I know y’all have said, in the past, that you would prefer not to reference files outside of the folder’s scope but I figured I’d throw out this proposal and see what y’all thought.

Proposal

What it does

Allows users to include files in their folder’s .stignore file using absolute paths.

Why people might want it

Makes it easier to manage your “global” .stignore files. No more ../../../../ in your .stignore files.

How it works

A new keyword #includeabs is available for use in your .stignore. When Syncthing processes your folder’s .stignore and encounters #includeabs it treats the string after #includeabs as an absolute file path for another .stignore-formatted file to load.

What it might break

If implemented correctly this should not break anything.

Who will implement/test it

Me.

Example Scenario

Setup

  1. I have an ignore file located at ~/foo
  2. I have a Syncthing folder located at ~/Sync/Bar
  3. I have an ignore file at ~/Sync/Bar/.stignore

I want Syncthing, when processing my folder at ~/Sync/Bar, to load the ignore patterns in ~/foo.

Without this proposed change ~/Sync/Bar/.stignore must contain the following line:

#include ../../foo

With this change an alternate version of that line could be:

#includeabs ~/foo

which is, imo, much more readable.

Thoughts?

@audriusbutkevicius’s response:

I think we should go the opposite way and prevent including anything from external directories. We permit that file to be a symlink so people can use that. Supporting includes outside of the folder causes pain with filesystem abstraction.

@harrisonhjones’s response:

Thanks for the response. I’m personally not a huge fan of symlinks; coming from a Windows background symlinks are still things I don’t naturally understand and I would not be surprised if others felt the same way. Additionally, on Windows there is pretty aweful support for symlinks, at least at this point.

For my own understanding can you go into a bit more detail on

Supporting includes outside of the folder causes pain with filesystem abstraction.

Why does it cause pain? How?

@audriusbutkevicius’s response:

Every syncthing folder has a “filesystem” where the filesystem is rooted at the root of the folder. Meaning that accessing anything out of the folder should crash/fail and burn, this is to avoid various attacks such as breaking out of the folder by time based symlink traversal attacks and so on.

Supporting loading ignores from random parts of the filesystem suddenly makes this isolation code an unbearable mess that opens up potential attacks, which I’d rather kill than expand.

Also, I think this should live on the forum for starters as there is nothing actionable here for now, so no reason for this to be an issue on github.

@calmh’s response:

Yeah there’s that thing, which is difficult to correct now for historical reasons. Regardless supporting absolutes in the stignore is indeed a step in the wrong direction so I’m going to veto it for that reason. Long term I still think we should get rid of ignore patterns in files as a concept and have some more elegant solution in the config/database with optionalish sync of patterns between peers.

Going Forward

so it seems that the core maintainers are against the #ignoreabs which is fine. However I’d still like to find a better solution for managing “global” ignore patterns. What are people’s thoughts? Maybe we allow you to define different ignore patterns in Syncthing and “apply” them to folders? Ie, in the UI somewhere you click “Create New Ignore Group”, add patterns, and then check which folders to apply them to. What do people think?

1 Like
2

Also relevant threads:

And probably others. I’m not going to link the next gen ignores issue…

3

I think that a potential solution for this would be to move ignores out of files, potentially to config/database/something else, then you could have a global one for all folders, and private one for individual folders. I guess then you’d have to come up with a creative way how to replicate the current “sync the ignores”.

4

potentially to config/database/something else

that’s what I was thinking. What if we modeled ignore pattern sets or groups in a similar way to the way we devices or folders. One way that some people might like:

Define a specific folder in Syncthing, let’s call it “Ignores”. It can contain any files you want but, on startup and file change, Syncthing will scan that folder for any files ending in .stignore (but not including just .stignore) and store the file names. Now a user can go into their Syncthing folders and, in the “ignore” tab of the UI, select which .stignore files in “Ignores” they want to apply. Which files have been selected is saved with the folder’s configuration.

If you wanted to sync this “Ignores” folder you could as well.

5

That’s still patterns in files on disk, with some sort of meta folder around it. We’re thinking about not having that but storing the patterns in either the config or the database, and perhaps communicate those to other peers - but not as files.

Regardless of how it’s done we need to consider the chicken-and-egg problem; that is, when you set up a new device and also a new folder, and it should have ignore patterns. If the patterns come from the other peer it might be too late as they will not have been known when we did first scan etc.

6

I think there are actually two problems here:

  1. I want to reduce the amount of hassle required to setup a new folder and keep its ignore patterns in “sync” with a master set of ignore patterns.
  2. I want to sync my ignore patterns with the folder itself.

Lots of ways we could solve this. One way:

  1. To solve the chicken and egg problem: sync the folder’s .stignore first and then recalculate what needs to be synced. Seems like a hack.
  2. Implement the “Ignores” directory I mentioned and let a user, in the UI, click “Overwrite” to overwrite a folder’s .stignore with the contents of the selected Ignore.

I’m trying to find a balance with a solution that adds value, re-uses existing patterns and functionality, and doesn’t feel too “hacky”.

7

@calmh @AudriusButkevicius so how do we move forward on this? How can I help? I don’t have a ton of side project time but I love Syncthing and would like to help make it better.

8

There are so many opinions and options around this that I think the first step should be to create a concrete proposal and get buy-in on that.

9

Great! How would you like me to go about doing that? I tried on that GitHub issue but maybe that wasn’t quite the right way…

10

I would suggest something more dynamic / amenable to change – a gist, or the wiki on syncthing/syncthing – with discussion here on the forum. Speaking just for myself, I don’t think I’ll be happy with a proposal centered around files on disk.

11

I can do that sure. A wiki +forum post seems to make the most sense.

Quick question: Is there a reason we don’t support syncthing the .stignore file itself? As someone familiar with Git I feel at home with .stignore but don’t quite understand why we don’t sync it (and why people hate it).

12

Because if I want to ignore my top secret government document folder, you’d know that I have one.

As @calmh said, anything revolving around files will probably not fly.

13

Ignore patterns are very often device local and not the same for everyone sharing a folder. To do that you’d have to ignore .stignore and at that point even I’m confused about what’s going on and what’s going to happen when it gets unignored or if someone included a file that ignored .stignore but my .stignore ignored that included file and … yeah.

14

FYI and slightly OT This might not help with your problem regarding global ignore patterns, but this will certainly help with creating and managing hardlinks, softlinks and symlinks with Windows Explorer! LinkShell : Windows Explorer shell extension for symlink support It’s just been updated yesterday, so it’s still maintained.

15

@harrisonhjones I try to understand your use-case. Why do you want to include ignore files outside of the shared folder? How does it save your time or give you practical flexibility? :thinking: As Syncthing is not a multi-user sync, you can’t force it on teams anyway. And setting up ignore pattern on two different computer once shouldn’t be a too big hassle?

For me I would like to see:

  1. the patterns to follow .gitignore logic (more on this issue here).

  2. Included ignore files should be relative to their own path (instead of from the sync-folder parent) → or .stignore should be placable like .gitignore everywhere (they are applied automatically) → no need for include statements

  3. To have a dropbox-like GUI to select and unselect folder (and eventually files) that would be written to the main-folder .stignore file image

1+2 together can be simplified by: “stignore should behaves similar to gitignore”.

16

I try to understand your use-case. Why do you want to include ignore files outside of the shared folder?

My original motivation stemmed from having a bunch of synced folders, all with the same .stignore. I wanted a way to keep all them “in sync”.

That said I have recently settled on a set of ignore patterns that work for me and haven’t needed to change them in a while so while it would still be nice to be able to keep them in sync it’s not a big deal anymore (for me)

  1. the patterns to follow .gitignore logic (more on this issue here).
  2. Included ignore files should be relative to their own path (instead of from the sync-folder parent) -> or .stignore should be placable like .gitignore everywhere (they are applied automatically) --> no need for include statements
  3. To have a dropbox-like GUI to select and unselect folder (and eventually files) that would be written to the main-folder .stignore file

#1 and #2 are potentially easy wins. #3 would be really cool but perhaps fairly difficult/time-consuming to implement.

18