Is it possible to see the "effective" ignore patterns?

I have an .stignore file with an #include rule and am not 100% certain it’s taken into account. Is there a way to see the “effective rules” for a repository? Or, the rules “as seen by syncthing”?

Currently the interface only says: “Ignore Patterns: Yes”, which is not particularly useful.

It might also be helpful to somehow see a detailed status of a repository, to see which files are ignored. Similar to something like git status -s output.

You can inspect the HTTP call responde with developer tools in chrome.

Alternatively check the docs for the rest api which will show you patterns.

Aha. This confirms my suspicion. It seems there’s something wrong with my ignore rules. This is the content of my .stignore:

 #include ~/.stignore

 *.pyc
 *.pyo
 .git
 .vagrant
 __pycache__
 env

And this is what I get as result from the REST call:

{
  "ignore": [
    "#include ~/.stignore",
    "",
    "*.pyc",
    "*.pyo",
    ".git",
    ".vagrant",
    "__pycache__",
    "env"
  ],
  "patterns": []
} 

Shouldn’t patterns contain something?

It’s becoming a bit of a nuisance, because it keeps syncing my .git and .vagrant folder. And as you can imagine, that wreaks a bit of havoc with development. I’ve had to recover those folders already twice now. Not too difficult, just a bit annoying :wink:

Pretty sure tilde isn’t expanded in that include. Probably we should print an error for that somewhere.

But the fact that patterns are empty is suspicious. Did syncthing manage to run a scan yet?

I think we error out on the include and then don’t parse anything. But we silently eat that error, as we expect errors from loading missing ignores and whatnot…

I have now replaced the tilde and get the same thing:

{
  "ignore": [
    "#include /home/exhuma/.stignore",
    "",
    "*.pyc",
    "*.pyo",
    ".git",
    ".vagrant",
    "__pycache__",
    "env"
  ],
  "patterns": []
}

To answer @AudriusButkevicius, no, I don’t think a scan is run. It is now continuously scanning. I believe the trouble is that on first run, I forgot to add my .git folder so it scanned all folders in all their glory. I have now a total of 362820 items in that repository :open_mouth:

Most of those files come from virtual environments, revision control or cache files which I should ignore anyway.

Those patterns are all in my “global” ignore file (the one I included). But as that errored out due to the tilde all those files got included. Now ST seems to be busy rescanning all the time, and I can’t get it to work on anything else.

I was already thinking about nuking syncthing and reinstall it again to start from scratch. But I wonder if there is not a less radical solution :wink:

I doubt a reinstall would have any effect. However, you can run with STTRACE=ignore to see what the ignore package is seeing. Does that /home/exhuma/.stignore exist and is readable?

You’d probably want to nuke the index directory on all sides to remove the now ignored files from index to cut its size down.

@calmh STTRACE=ignore does nothing. It’s btw also not mentioned on http://docs.syncthing.net/dev/debugging.html Has this been added in any of the latest releases?

@AudriusButkevicius Where can I find the index file?

You’re right, there’s no ignore trace facility, my bad. It’s part of the model. That code does emit a warning on errors from loading ignores, except if it’s “file not found”. In this case I suspect the error is file-not-found (again, does that included file actually exist?), it’s just not the file-not-found that we expect…

Oh. Sorry. Forgot about that info :slight_smile: Yes, the file exists and is readable.

I am now running using STTRACE=model. I have not yet seen anything come up regarding an ignore error. What I noticed however is that messages flew by concerning ignore patterns on other folders (no errors, just the debug message:

DEBUG: rwFolder/data_files@0x1d6e7180 ignore patterns have changed, resetting prevVer

This message is not showing up for my “work” repository.

Right. I reproduced it. The include path is always interpreted as folder relative, regardless of it looking absolute. This may be a bug or not, depending on viewpoint. You may be able to escape the folder with an appropriate number of ../ at the start.

Hence we do get a file-not-found for that include, and abort loading ignores without printing an error. That is a bug for sure.

Yes. Using a relative path did the trick! Thanks.

To me it feels a bit counter intuitive to not handle a filename as not absolute even if it looks like it. But I understand the thought behind it. I assume you consider the beginning slash as an anchor to the root of the repository instead of the OS. Which is fine.

I think the biggest issue is the silence :wink: If you’d get some kind of pop-up in the web-interface everybody can figure out the rest on their own :stuck_out_tongue:

I’m still waiting for the node to come up. It’s currently showing the state of each repository as UNKNOWN.

A question: what happens to the files in the index? I mean, they were not ignored up until now, so they were added. Will they be removed from the index now that they are?

Yeah, that’s a bug.

They’ll stick around, but syncthing will pretend not to see them.

Hmmm… in that case it might still be of use to somehow start over. It seems that the rescan still takes ages, even though the ignores are now honoured (at least I think so).

I will keep it running for a while and see if everything works fast enough. If it turns out that it’s still too sluggish I might need to start over.

Bad news…

I just checked the web-interface again and it shows about 80’000 files out of sync. On a quick glance into the list, it also shows files from inside the .git folder. So the ignores still don’t seem to be taken into consideration. However, looking into the REST API output, it looks fine:

{
"ignore": [
    "#include ../.stignore",
    "",
    "*.pyc",
    "*.pyo",
    ".git",
    ".vagrant",
    "__pycache__",
    "env"
],
"patterns": [
    "^[^/]*\.bak$",
    "^.*/[^/]*\.bak$",
    "^[^/]*\.bak/.*$",
    "^.*/[^/]*\.bak/.*$",
    "^[^/]*~$",
    "^.*/[^/]*~$",
    "^[^/]*~/.*$",
    "^.*/[^/]*~/.*$",
    "^\.[^/]*\.swp$",
    "^.*/\.[^/]*\.swp$",
    "^\.[^/]*\.swp/.*$",
    "^.*/\.[^/]*\.swp/.*$",
    "^\.git$",
    "^.*/\.git$",
    "^\.git/.*$",
    "^.*/\.git/.*$",
    "^\.vagrant$",
    "^.*/\.vagrant$",
    "^\.vagrant/.*$",
    "^.*/\.vagrant/.*$",
    "^env$",
    "^.*/env$",
    "^env/.*$",
    "^.*/env/.*$",
    "^[^/]*\.pyc$",
    "^.*/[^/]*\.pyc$",
    "^[^/]*\.pyc/.*$",
    "^.*/[^/]*\.pyc/.*$",
    "^[^/]*\.pyo$",
    "^.*/[^/]*\.pyo$",
    "^[^/]*\.pyo/.*$",
    "^.*/[^/]*\.pyo/.*$",
    "^\.git$",
    "^.*/\.git$",
    "^\.git/.*$",
    "^.*/\.git/.*$",
    "^\.vagrant$",
    "^.*/\.vagrant$",
    "^\.vagrant/.*$",
    "^.*/\.vagrant/.*$",
    "^__pycache__$",
    "^.*/__pycache__$",
    "^__pycache__/.*$",
    "^.*/__pycache__/.*$",
    "^env$",
    "^.*/env$",
    "^env/.*$",
    "^.*/env/.*$"
]
}

Any ideas?

I think you have to restart in order for the out of sync to disappear after the patterns actually started working.

I’m sorry this has happened to you but I’m glad to have confirmation that syncing git repos is a BAD idea.

@AudriusButkevicius You mentioned cleaning only the index. How do I do that? Is it only a file to remove somewhere?

@lfam Weeellll… .git should be quite resiliant to this (due to how object hashes work). But I still would not recommend it. I think I got lucky in my case. By far the biggest annoyance was vagrant. The .vagrant folder contains machine meta-data which is only valid for one host. If that gets synced you need to recreate the VM (an manually go into VirtualBox to delete the old one). Which can take a lot of time.