I’ve been using Syncthing across linux/android/windows for a year and have been loving it. I recently spun up a VM to play around with a few things and decided, why not install Syncthing and back up the entire home folder?
I did this:
Install Syncthing as root (i.e. in /root/)
Set up a systemd service to run syncthing at boot
Added the root of the drive (/)
Set ignore patterns as the following two lines:
!/home
*
With the above ignore patterns, I’d expect Syncthing to wake up, look at the root, ignore every single folder in there except home, and be happy.
However what I see is that I get “Filesystem Watcher Errors” telling me syncthing can’t find certain files such as:
[hostname] lstat /proc/2451/fdinfo/27: no such file or directory
Why is it even looking in /proc? Doesn’t /proc match the second line of the ignore file? Doesn’t this mean it will just move on to the next folder? With the above ignores I expect syncthing to only look in /home and just gloss over all other folders.
More info
Syncthing is running as root
I’ve updated the number of “max_user_watches” to 204800 and restarted
As soon as there is at least one exclude in the ignore file, Syncthing will scan everything, as the exclude can now also be e.g. !*.pdf to sync all pdf files. There is no optimization for ignore files like yours, where it is obvious, that only /home should be synced.
Yes, my first instinct was to just sync /home but I figured I’d do the entire root so I can selectively choose certain things (like /var/www or various config files in /etc) by simply adding them to the exclude file with a “!”.
I’m treating the exclude file as a “whitelist” where I specifically list all the folders I want it to include but end with a line that says “ignore all else”.
The *first* pattern that matches will decide the fate of a given file.
I assumed this meant it would scan files/folders, applying the pattern to each one of them before descending into any folders. For example, this would mean that it would see /proc because it is indeed a folder on the root. It would then see that /proc matches the * exclude and exclude the entire folder, moving on to the next folder.
You mean to tell me that instead, it descends into /proc? Wouldn’t anything it finds in there be invalidated by the fact that /proc itself is ignored? If I put a specific ignore for /proc, will it still go INTO /proc? It sounds like it will? What’s the point of ignoring entire folders if they aren’t really ignored? Shouldn’t a parent folder being ignored take precedence over a child (either another folder or file within that folder’s tree) that possibly matches a pattern?
Wweich explained how it works. Things could be optimized in the case when all !-patterns are absolute and rooted, but that’s not currently implemented.
Given that you probably have a limited number of directories in the root, listing them explicitly and not having any !-patterns should do what you want;
/etc
/lib
/var
...
# /home not mentioned here
This is unambiguous enough that Syncthing won’t traverse /etc etc.
Now that I understand it will scan everything regardless, I see why the error is occuring! I’ll either explicitly add specific directories to the ignore list or just sync the /home folder only. Thanks!
!FOO is no different from !*.pdf and the latter forces you to traverse down into every directory, even if they are ignored, regardless of the order in which they are defined.
Yes, this could be optimized, but as usual, PRs welcome.
Now I know, why syncthing is taking ages and heaten up my harddrive, while scanning some of my folders, where I use same ignores as the OP. Only on Mac or Windows I don’t get error messages, so I never had the luck to identify the cause.
Lets cross fingers for PRs to come (whatever PRs means…)
