Allowing to sync the .stignore file - any caveats?

Development
1

First of all, I am not trying to start any discussion on why and why not synchronisation of the .stignore file should be allowed. The whole thing is just for my personal use :sweat_smile:.

I have modified the source code and removed .stignore from the hard coded internal file list.

--- a/lib/fs/filesystem.go
+++ b/lib/fs/filesystem.go
@@ -204,7 +204,7 @@ func NewFilesystem(fsType FilesystemType, uri string) Filesystem {
 // path must be clean (i.e., in canonical shortest form).
 func IsInternal(file string) bool {
        // fs cannot import config, so we hard code .stfolder here (config.DefaultMarkerName)
-       internals := []string{".stfolder", ".stignore", ".stversions"}
+       internals := []string{".stfolder", ".stversions"}
        for _, internal := range internals {
                if file == internal {
                        return true

The modified code works, resulting in .stignore being synchronised as any other file.

I would just like to ask - are there are any possible caveats, except for when using relative patterns with #include linking to files outside of the folder in question, or when self-ignoring .stignore with patterns like *?

2 Likes
2

Self-ignoring is the first one that struck me, as that will be difficult and annoying to undo once you’ve done it. The other thing which is mostly related to #include is that when the ignore file can’t be loaded due to some error the folder is stopped. So if you introduce an error you’ll stop the folder on every device, and need to go around to each device fixing it.

1 Like
3

This is what I have gathered so far:

  • Self-ignoring: The ignored .stignore is no longer sent to other devices and works only locally (i.e. as it does in a normal Syncthing installation). Once unignored, it is synced again with no problems.

  • Broken #include: If the included file does not exist on other devices, then the folder synchronisation is stopped there, which requires a manual intervention to recover, so one needs to be very careful here.

  • Unhidden .stignore: The synced file loses its hidden attribute in Windows.

4

The third point is being fixed in a current PR.

A fourth point is that there is no opt-out to have my own ignore patterns if you sync it.

A fifth point is that this setup won’t make sense when you add a new machine to the cluster. It won’t have the ignore patterns when it does its initial pull (but ideally the other devices won’t announce any ignored files if they have the same patterns) nor when it does an initial scan of any pre-existing content.

2 Likes
5

Thank you. I have no checked the PRs recently. I have been testing only with v1.4.2 stable, so I also do not have any of the newer commits applied.

It seems to be possible, as long as you self-ignore .stignore on the device in question. The patterns are still applied locally, regardless of whether the .stignore file itself is ignored or not.

Yes, that is true. In my case, however, I mostly use my ignores in a few already established folders. I have also been updating my ignore patterns over time. By syncing .stignore like this, there is no need to go over all of them and update the ignore patterns manually any more.

Well, to tell the truth, until now I have actually synced my ignore patterns using additional #include files, but the problem with them is that they still require manual configuration when setting up the folder for the first time (and especially when the folder already exists and is not empty).

1 Like
6

I have decided to put a halt on this one, as there is really not much difference from using #include, with the exception for the initial folder setup, and with relative paths #include can also be used to share a single ignore file between difference folders.

What I would really like to do is either to

a) be able to set a custom name for .stignore for each folder

or

b) use a custom folder marker, and then put .stignore there

so that I could share the same folder multiple times with different ignore patterns. This is important for me, because otherwise I need to set massive ignore patterns on the receiving sides, and those take up both disc space and also drain battery. I have already had Syncthing crash on one Android device with a very small /system partition because of that. Right now, I have split the files into multiple folders, which are then shared separately to different devices, but this has been getting messier and messier with time, so I would strongly prefer to keep them condensed.

I have tried to modify the source code at least a few times, but failed so far. I guess that I will eventually have to learn the basics of golang for real to be able to achieve it :wink:.

1 Like
7

Both of those are kind of reasonable. I had a branch or just issue or something filed to move ignores into the marker directory, since that’s what it’s for. There are migration difficulties, like how do we interpret existing #includes and such. Maybe we should try multiple locations in order ($marker/ignores.txt and then .stignore if that doesn’t exist, or some such).

2 Likes
8

Yes, with my very limited skills I only managed to move .stignore into a hard-coded .stfolder, without using the $marker or worrying about backwards compatibility, but it did not really help with anything (except for reducing the visual clutter a tiny bit), so I eventually gave up.

9

I’m somehow dreaming of having .stignore at /folderroot/.stfolder/.stignore . Because: we already can configure the folder markers name and I could then share the same root folder two times as two different folder id’s. That would give me the opportunity to rename the folder markers of both like .stfolder1 .stfolder2, each having a .stignore in it. In this case I could get rid of the junctioned directories on Windows by sharing the same folder two times with different local ignores.

2 Likes
10

Issue is up for grabs.

2 Likes
11

Hi Synchers I have 2 questions about shared patterns (#include or other way).

Let’s say I use #include $FOLDER/some-path/someFile (utf8, I read the doc).

As long as the option Auto Normalize is left set to its default yes can we seamlessly use any of the directory separators / or \ in patterns in a mixed Win/Nux cluster ?

The other question raises about when editing the said someFile: when is the change taken into account by devices that rely on the file? Special scan? Wait for normal FS watch? Wait for full rescan? Checked on Folder/Device/Machine resume ?

I ask because I have a 60GB folder with ~150k files that hogs mechanical consumer grade disks and for them I had to change Rescan Interval to 12h (and disable restart on Wakeup for laptops that go to sleep).

Thank you

12

Auto normalization has nothing to do with file separators, that’s about encoding. For ignore syntax a path separator is always a slash (/). Wrong, see later posts.

Whenever there’s any scan or sync/pull, the ignore patterns are checked for changes. And a change to an included, synced file is like any other change, i.e. it will get picked up by watching and periodic scans.

13

Thank you Simon Glad for the second answer :slight_smile:

So there is something puzzling me: when I first tested the #include keyword in the GUI (Windows) I used \ for the separator and I made a mistake in the path to the file (forgot a dir level), which ended in error (approx “could not load file”). So I went to .stignore to check, found my error and everything went OK but some files that would be ignored were synced in the lap time, maybe I forgot a pattern. But I had both the path to file and the patterns with \ and it worked (in fact there is no Linux in this cluster, it’s for a friend personal use, pure Windows until now). I asked the question for my own mix use. After your answer changed both path and patterns to / and it seems* to work too. So is there a mechanism that transforms any / or \ we use to fit to the OS?

*not so sure: a single pattern with the \ and I can’t remember if I created the related dir everywhere before.

14

For the path given to #include both slashes and backslashes are indeed ok. For patterns it must be slashes though. There’s no special mechanism, it’s just that the go package used to handle the path given to include can manage both. Wrong, see later posts.

1 Like
15

Thanks a lot, this is the ultimate answer I hoped.

This is worth a Note in the doc!

16

Agreed, but for simplicity’s sake I’d just state “slashes (/) only as directory separators” in ignore patterns. Adding any distinction between include and the rest is just unnecessary complexity that is a source for confusion. Wrong, see later posts.

And as usual, please consider contributing any improvements you’d like to see directly to GitHub - syncthing/docs: Documentation site :wink:

17

I’m pretty sure backslashes as separator in ignore patterns is fine on Windows?

2 Likes
18

They are. I personally use / just for consistency between different operating systems, but \ does work on Windows too.

19

Right, one of the first things done to the input is normalize the separators, so both types work. Thanks for correcting (edited above).

1 Like
20

I changed my mind, and several weeks ago added the commit to my personal Syncthing build. Since then, I have been syncing .stignore like any other file.

It actually works great. There is no need to manage ignores on each devices separately any more, and if I do need to do so, I can just add .stignore to the ignore list on that machine, which will then make that ignore list local (i.e. mimic the default Syncthing behaviour).

3 Likes