un-ignore subfolders & files of ignored folder

Feature
1

As mentioned in the ignore patterns discussion, ignored folders are not searched for subfolders or files that match an “un-ignore” pattern. However, there are valid use cases for this to be changed. In my case, a work folder is synced between 2 of my PCs, with sub-folders for various projects. I would like to be able to share these with collaborators when necessary. Installing Syncthing for them & setting up an ignore pattern like

!/projects/project1/ *

would be neat! Moving the project folders to a higher level setting up separate sync, would be possible, but at disproportionally more work. Esp. as collaborators may need access to more or other directories. If Syncthing would search ignored folders for un-ignored sub-folders and files, I could simply send them the lines to copy into their ignore patterns to “unlock” a project folder, or a file type.

Maybe this could be done minimally invasive by adding an ignore pattern modifier to force this “un-ignoring” behavior, or by using the line numbers as a ranking like

* // ignores everything

!/projects/project1/ // un-ignores this folder & therefore searches previously ignored ones for it

or

!/projects/project1/ // ranks un-ignoring this folder higher & therefore searches subsequently ignored ones for it

* // ignores everything, but with lower priority

In either case, please consider implementing a mechanism for this behaviour. Thank you!

2

Including a subdirectory explicitly is possible:

!/projects/project1
/projects/
!/projects

*

I think will do what you want but some might find the syntax is, well, a little ugly.

The [ignore patterns discussion] (https://forum.syncthing.net/t/excluding-files-from-synchronization-ignoring/80/21) cites [.gitignore] (Git - gitignore Documentation), which rather puzzled me as I didn’t see git and syncthing as comparable.

I’ve used git a fair bit but never have had any trouble setting up ignores rules like I have had with sycnthing and I have never used the ! syntax to ‘un-exclude’ something. So I read the reference. I’ve always used .gitignore files ‘within the repository’ and I can see how the ! syntax could be useful. However, I’ve never really used gitignore rules ‘outside the repository’, which is, I guess, analogous with the syncthing use case. I see that as a very different use case and ! syntax as problematical.

Is syncthing trying to emulate gitnore ? Sadly not. Among the differences are that with github the ! syntax “any matching file excluded by a previous pattern will become included again” so the ! rule appears second, whereas with syncthing it must appear first.

Another difference that I am not the only one to find confusing is the interpretation of name/. Plain name matches either a file or a directory. To gitignore, and to me, name/ matches a directory but not a file. However, in syncthing name/ matches the contents of the directory but not the directory itself. It is the same as name/* :scream: ) . The difference is subtle:

!/projects/project1/
/projects/*
!/projects/

*

does not mean at all what you might think it should (even though to my mind it expresses the intent more clearly).

Can the (first) example be extended to second and third level directories ? Yes but the tedium increases with every level:

!/projects/project1/workpackage24/task7
/projects/project1/workpackage24/
!/projects/project1/workpackage24
/projects/project1/
!/projects/project1
/projects/
!/projects
*

This works but I spent an entire weekend setting up my .stignore file to include what I wanted and exclude what I didn’t want and test the damned configuration.

If I ever have to do this again (and I really hope I don’t) I’ll write a script to pre-process a .stignore file from one that has simple (one-line) include rules into one with slews of rules required by syncthing.

In the end, software is so much better at this kind of thing than we are and you just have to wonder why syncthing can’t do it for us.

2 Likes
3

I see that this could do with some improvements. I suggest you raise an issue on github as an enhancement, but please search for a simillar issue incase one already exists.

4

Indeed. This is probably just a matter of getting the desired behavior nailed down in tests, and then we can fix it.  :)

5

Hi,

Thanks you each for your one line of encouragement. I take it you are busy people.

I’m not sure what, if anything, you are asking me to do.

I think syncthing is an interesting piece of software that syncs files very well but I’m not sure if you want it to be a system tool or an application for friendly users (or both).

Which use cases you want to support has a bearing on what I might propose and whether it would be worth while.

syncthing isn’t a backup tool and it isn’t git and drawing analogies with them only takes you so far. I think syncthing has a symmetry that backup programs and git don’t have and I believe it merits an inclusion/exclusion mechanism that reflects this.

Use Case: I’m Not A PC

For my first trial I tried the default: sync everything. This probably works very well with the defaults on most PCs and mobile devices.

For me, syncthing got stuck. I learnt that I’d been doing backups relying on read access only for years but that syncthing insists on read and write access to files (at both ends). Symmetry.

For me, a system tool should not require me to reorganise my file systems in order to use it. I should be able to configure it to work with what exists and take a view on how to follow up.

The ‘ignore’ feature was the only way forward. I have cases where it was easier to say what to include rather than what to exclude but the current include syntax is clumsy.

I can imagine easier ways to handle files can’t be synced.

Use Case: Projects on a File Server

This is Cathryne’s use case. One sync folder on a file server and client PCs that sync only those projects their owner’s want.

This is a use case I can easy see myself wanting to use.

The current syncthing alternative would be to set up a sync folder for each project. That would seem to me to imply a lot more of Cathryne’s time taken up in administration.

Use Case: The All You Can Eat Buffet

I had a brief encounter with bitsync. The sync was very badly set up. This wasn’t the tools fault. The project manager’s view was it only needed a few clicks to set up. What could possibly be wrong with that ?

The sync was used to distribute (development) releases. Each release was another subdirectory and there was a release every other week. Each release was the order of 5 Gb.

What a waste resources.

It was a waste of time too: someone new to the project had to wait while all the releases synced because there was no way to prioritise the synchronisation of the most recent release.

The general point here is that, when joining a large, existing sync, it is a lot easier to list the things you do want than it is to list all the things you don’t want (and may not know or care about).

Use Case: Divide and Conquer

This use case is mine but is it similar to Cathryne’s but there is a twist.

I have a file system with ‘projects’, just as Cathryn does, but I have ‘projects’ within ‘projects’, as it were. Hence the tedious syntax illustrated in my earlier contribution.

Roughly everyone has read access to everything … there are no secrets … but only one person has write access … different people are responsible for different ‘projects’.

Currently that would mean a separate instance of syncthing for each user … OK … which means a sync folder for each user … oh dear …

With the current ‘ignore’ scheme each user needs a different set of rules but there can be only one .stignore file in the root directory of the file system.

syncthing is not git

Committing files in git is a way of backing up files. Pushing and pulling git repositories is synchronising backups.

It’s not syncthing. It’s not even close.

The .gitignore file (as the documentation says) is about ignoring ‘untracked’ files. It has no effect on which files are pushed or pulled and it does stop you committing files that are ‘ignored’ if that is what you really need to do. It just makes it easy not to commit files that have not been, and never should be, committed.

The patterns in .stignore represent not just files that should not be synced ‘to’ other devices but also files that should not be synced ‘from’ other devices. Scary and intriguing.

Where am I going with this ?

I think inclusion and exclusion should be on an equal footing: the current ignore with the ! syntax is unequal.

A simple analogy might the hosts.allow and hosts.deny mechanism on Unix systems. If you don’t know or like this analogy, pick any firewall you are familiar with.

You can start with “allow everything” and add a few exclusion rules. You can end with “exclude everything” and insert a few inclusion rules. You can do anything in between.

Perhaps such a scheme would require too much testing.

Assuming that it is something you might consider, the first task would be choosing a syntax that might allow enhancements to be added later. The ! syntax is no good.

At present, if two devices have different .stignore files then one or other will never achieve 100% synchronisation. Very disconcerting. The more devices in the sync, the more disconcerting.

With the right syntax to start with, an enhancement might be new syntax that means ‘when device equals’. This would allow the rules to be tweaked until 100% synchronisation is achieved with each device.

Another enhancement might be syntax that means ‘when user equals’. This would allow rules in the same file for different instances of syncthing running under different user ids but with the same sync folder.

There might be, at some time, a case to be made for distinguishing ‘in’ rules from ‘out’ rules. The ‘everything else’ rule might be an example: sync everything else ‘out’ but nothing else ‘in’.

6

I think you should just use sharepoint or something that has this granularity of control. I don’t think syncthing will ever work on a per user level.

As to all your points made, I still don’t understand why the ! syntax doesn’t cut it. I know that currently it doesn’t work as expected in the cases described in this post, but with a few tweaks it should, and when it does, what is the use case that it doesn’t cover? (Apart from this multiuser thing which is already solvable with the #include directive in .stignore)

7

The above issue should be solved by:

But this still relies on the order of the ignores, so exclusions must come before inclusions.

1 Like
9

This is actually harder than we though (because exclusions can contain **, * and not be anchored to the root making it very hard to figure out what folders are needed in order to be able to sync the exclusion)

But it now has a ticket, and is assigned to the v1.0 milestone.

10

But it now has a ticket, and is assigned to the v1.0 milestone.

Thanks for trying. I can imaging that the ** notation would give you headaches.

As to all your points made, I still don’t understand why the ! syntax doesn’t cut it.

Well, it sort of goes like this … I’m a user (of syncthing). I think of it as syncing files. Right ? So when I see the ! I read “Do not not include the files that match this pattern in the sync”. I was brought up not to use double negatives (at least not in English). Your response could well be that the foibles of natural languages is why we use symbols to mean just what we want them to mean. As a programmer, I get that. But I understand that the whole excuse for GUIs is that the average PC finds programming languages arcane and needs something friendly without all the symbols. So while I think the ! (and the **) syntax are ‘cool’, they don’t really belong in a user friendly interface and I try to get my desired result without them.

Apart from this multiuser thing which is already solvable with the #include directive in .stignore.

Please tell me more. I tried expression such as:

#include $USER.ignore
#include $HOME/ignore.these
#include ~/ignore.these

each to no avail (I try them one at a time). I must be missing something. What I did find is that, given #include with a path to a file that does not exist (or a pattern that is not understood like the above), syncthing appears to ignore ,stignore altogether and tries to sync everything. Not quite the kind of behaviour I was expecting (or wanting).

I think you should just use sharepoint

I appreciate the irony. If commercial software were an option, I might not be using Linux at all and sharepoint might be something I’ve done more than merely heard of. Much more likely I would never have even tried syncthing.

something that has this granularity of control

The granularity of control that I need is a program that will sync what it can and leave the rest to me to sort out later.

The first time I used syncthing (perhaps its behaviour has changed since), the sync got stuck because syncthing did not have r/w access to the directory lost+found. I could not tell what state the sync was in, which was worse than useless. All my fiddling with .stignore is about telling syncthing to ignore files it cannot sync anyway, pretty much on a case-by-case basis.

I would be happy if it synced all the files and directories it could a left me somewhere a list of those it failed to sync (that were not already excluded by a pattern in .stignore).

I don’t think syncthing will ever work on a per user level.

That is a shame. Semantics (enabled as an option) such as “walk the directory tree but only sync those file whose owner is the user id under which the syncthing is running” would probably mean I would not need ,stignore at all. No setuid, no root, just a plain, safe, mind your own business, user space application.

On a positive note, I’ve discovered that syncthing takes note of changes to .stignore without having to be restarted. I guess it ‘re-reads’ .stignore for each scan of the sync folder. I like this. Since the whole idea of sync thing is to cope with changes to the contents of the sync folder without user intervention at the GUI, it does seem entirely reasonable that it should also cope with changes to the .stignore without user intervention at the GUI.

Now, how to freshen ,stignore in time for the next scan ? What I need is to be able to ask syncthing to run a script for me before each scan. All I need in my script:

find . -not -readable -o -not -writable > .stignore

That would be a programmer’s solution to the problem. It might look ugly. There might be alternatives that use fewer resources but running a script would allow me to use sed or Perl or Python with regular expressions I’m familiar with but I would not need them: no ! and no **.

As it stands, I don’t trust syncthing enough to leave it running in the background. I have the rescan interval set to 0 (I like that feature) and tend to stop and start syncthing so I can watch only when I know I have the time to sort out a sync that gets stuck.

Perhaps what I need to do is run syncthing for each user with a largish rescan interval and then arrange for each user to run a cron job just before the scheduled re-scan to run find to freshen .stignore so that it will be correct for that user’s re-scan. Of course, I wouldn’t need that for every sync folder but, just in case …

Nah, there has to be an easier way.

11

The .stignore solution to multiuser I was talking about is:

You have settings/john and settings/jane in the folder. Now on Johns machine in .stignore you do #include settings/john on and the same equivalent for Jane.

Now these two files are synced hence you can remotely control their .stignore content for them from your machine.

12

I wanted to update this Ancient Dead Creature, as the solution to the title statement of the original post doesn’t seem obvious (to me at least), and I stumbled upon this from a google search.

I was backing up my VPS and wanted to get very granular with my set up. I just set this up on my machine.

In .stignore I placed the following

#include /home/user/folder1/rules.conf

Where the path is relative to the root of the sync share. Working with the whole VPS, I just entered the actual path.

It is good to think of building the ignore list file from the bottom up for this level of granularity. So start with an “ignore all” rule at the bottom.

*

Then, above that, tell syncthing to include the directory, home in this case, and all of its contents by leaving out the trailing slash.

!/home
*

Next tell syncthing to ignore the general contents of home.

/home/*
!/home
*

Now syncthing is ignoring everything but the actual home folder. To descend into a specific user directory we have to repeat the pattern.

/home/user/*
!/home/user
/home/*
!/home
*

If you want to add the entire user directory, just leave out that top line. To add a specific directory, and all of its contents, therefore, just add a line like so…

!/home/user/folder0
/home/user/*
!/home/user
/home/*
!/home
*

And finally to descend into a nested folder and sync only one file among many within it, just add the following lines at the top of the file

!/home/user/folder1/rules.conf
/home/user/folder1/*
!/home/user/folder1
!/home/user/folder0
/home/user/*
!/home/user
/home/*
!/home
*

So now the sync will get the file rules.conf in folder1, and folder0 and all of its contents, while maintaining file-system structure for user.

In this manner it is easy to add the specific appropriate folders and files in /etc, /var, and any other directory desired. Then set the folder to master, and on the receiving end(s) add in some basic file versioning.

If there’s an easier way to do this, I don’t know, it sure would be nice if there was a more intuitive way to include granularity as an option via a GUI. With the web interface, that would make syncthing simply ideal for server backup functions.

7 Likes
13

@joshp23 Thanks! This was not intuitive to me either.

14

Thanks ! This should deserve a pin in the howto of syncthing. :slight_smile: