So I’ve seen several related questions so far but haven’t managed to be clear about the outcome of “nesting folders” within a sync or set of syncs.
My usecase:
I want to back up critical parts of my intra on a backup machine, I set a sync for a number of relevant directories on the intra, but ultimately some were too large to justify backing up entirely, so I broke out the dirs that needed custom configs and ended up with a fairly large list of directories to back up independently, plus others from the system.
As such, I don’t want to deploy these directories independently on my new machine, so normally I would copy them on my backup machine into a “package” dir, delete all .stfolder files in the child directories and sync that, then copy relevant files on machine 2 after sync. But now the data has grown too large to copy on my backup machine - I would like to make the directory that contains ALL my individual backup syncs into it’s own sync, and sync that to my new machine instead of copy&sync.
That said I’m TERRIFIED of losing this data and my configs - is it safe to do what I want to do or do I need to just get more hard drives? Also does it cause some strange effect with versioning?
It may be relevant that once I deploy the parent sync to my new machine at /backup - I copy all the files into their respective end destinations and delete the /backup dir entirely and close the sync down - it’s used only to deploy - so I don’t care about the parent’s ability to sync and it won’t have versioning, but it’s child sync’s will and they WILL continue to back up those files/dirs on the intra thereafter(hopefully with accurate versioning) until I need to redeploy again someday…
Honestly, it would be nice to see some kind of a diagram of what you’re trying to do for better understanding, but I think that I can tell you as someone who’s tried doing nested folders previously what kind of problems you may possibly expect in general.
Syncthing internal folders and files like .stfolder, .stversions, .stignore, etc. are only special in the main folder root. This means that when you’ve got nested folders that are also shared through Syncthing, the parent folder will sync those internal folders and files too. To avoid this, you should at least add them to the parent folder’s ignore patterns.
You definitely don’t want to have both the parent and the nested folder shared between the same devices, as this will cause both of them trying to sync the same files at the same time, wasting resources and always leading to one of them failing. For this reason, if you share the parent folder between the same devices already, then you should not share the child folder separately. This also means that you don’t want to use the Introducer option, as it will automatically share all the folders without taking the parent/nested relationship into consideration.
Using versioning in both the parent and the nested folder at the same time is problematic. You probably only want to enable it for the parent folder.
In general, on the basis of my experimentation and experience, I would personally advise against using nested folders, as even if you do succeed in making them work without major issues, you will still need to keep an eye on them and always pay attention not to make any mistakes, which for myself was just too cumbersome.
So basically I would have parent and children on in Syncthing on the backup machine, parent being nothing but the directory that contains the synced children(no other files, etc), then I would sync that parent like a deploy package to the new machine, I would copy necessary files (data files and system configs - no Syncthing files will be transferred). After the files of interest are transferred I delete the sync from Syncthing UI on the new machine and remove the directory entirely (with all subsyncs inside).
So I really am only using that sync to deploy, then it goes back to “unshared” on the backup machine and serves only to aggregate the syncs it contains, but is never shared or synced outside of backup machine again.
I will then use my previous config.xml for syncthing, delete the old direct syncs from backup from UI and remove synced directories, reshare them fresh from intra as though it was the first time, and let them sync normally back up to the backup machine. Again versinoing will only be on in the new “subsyncs” and all old versioning data will be gone.
Can ignore patterns work more like “include this” or exclusively exclude? If I have to write exclusion patterns I’m going to have to exclude several hundred - my /var is HUGE with user data dirs, custom user applications and more - backing that all up is not feasible nor is trying to exclude all current and future unwanted contents of the top level dir I want to back children up for.
In other words, I need A LOT more excluded than included in /var/bin, can I use a positive add pattern somehow?
Yeah just did - so I was expecting regular expressions, but I don’t mind learning a different syntax/system IF it can do what I want - as I don’t yet understand “glob patterns” and the example in the docs doesn’t help clarify whether you can exclude all and include selectively, it only shows examples of selectively excluding(ignoring).
So I want to repose my question - is it possible to [exclude all] → [except /a], [except /b], etc? If not, there’s no purpose in me learning this pattern system.
I really do appreciate your time - just want to know if this is a feature before I spend the time to learn a new pattern matching standard…
in my parent dir’s .stignore ignore all dirs and subdirs except ./bin/tensor_flow? (your example provides no clarity about whether these control chars can be used together or how they would behave)
So in RegEx I might do something like:
^(?!bin\/tensor_flow|www).*$
Might look kinda like this if Syncthing used RegEx and it “stacked” like your examples:
^(?!bin\/tensor_flow).*$
^(?!www).*$
Which would select only files/dirs that didn’t start with bin/tensor_flow or www for exclusion, but I don’t see any equivalent to (?!) or whether something even CAN be paired with * since your examples show only single control character used with these “glob patterns”. Example/sample selection for clarity: RegExr: Learn, Build, & Test RegEx
That’s the only way I can see to positively select for ignoring, but since “glob patterns” have no equivalent(?) to RegEx’s ?! and other behaviors, and it’s not clear I can combine with * anyway (allowing me to select “everything except” to be ignored), can glob patterns stack asterisk first to ignore all and then selectively add exceptions using ! below that FOR ALL full AND partial “from beginning of line” matches? Will I really have to find out by testing live on my valuable data?? Will our hero finally find satisfaction in his attempt to back up his system???
Might be a nice feature if you could have groups or tags? Then a number of syncs could be shared to remote as a group or by tag rather than having to share them all individually and they keep the directory structure and naming convention from the sharing machine, for when you’re upgrading systems lets say and essentially need to move syncs “upstream” or have a different machine that needs most of another remote’s synced data… Just a thought…
We don’t do that here, but if you really want to, you can add something like “[SOLVED]” to the topic title if you’re willing to signal that the problem has been solved .
So that was fine in the package I had created, but when I tried that at root dir (/) to start my long-term syncs, Syncthing got REAL wacky. The impression I get is that it still needs to scan the entire root despite having .stignore that largely excludes most of it? Not sure but I got inotify “can’t handle that much” error message during most of scan and persisted after completing and it’s results were DRASTICALLY larger than they should be…
So not sure exactly why but I’m not shocked that trying to run a sync on root dir, even with exclusions wasn’t a great plan. So now I’m wondering, if I turn off “Watch for Changes” and scan only on schedule:
The files will be updated during scheduled scan correct?
Can standard file versioning still work normally (albeit at an interval)?
That will eliminate the use of inotify entirely (or at least solve the error)?
Note that !etc will match not just the root of the Syncthing folder, but also any subdirectories. So if the base directory is /, at a bare minimum, the following two directories will be included:
/etc
/usr/local/etc
(Use the pattern !/etc if only /etc should be included.)
Because your ignore pattern has include patterns, per the Ignoring Files page:
Include patterns (that begin with !) cause Syncthing to traverse the entire directory tree regardless of other ignore patterns. If the watcher is enabled, the entire directory tree will be watched as well.
Yes, there can potentially be quite a few inotify watches if the Syncthing folder is set to /. Check the value for the following kernel setting:
/proc/sys/fs/inotify/max_user_watches
Default value is typically 16384, but could be higher or lower depending on the particular distro. inotify doesn’t traverse subdirectories, so a separate watch is required per directory.
Correct.
Yes.
When using the file watcher, each time a change has been detected, Syncthing initiates a scan (after a default 10 second delay), so it’s like the interval scan but targeted and not on a fixed schedule.
Yes, if there’s only one Syncthing folder, or file watcher is disabled for all folders.
If your system has the memory to spare, increasing the value for max_user_watches will also resolve the error (see the inotify manpage for details).
I don’t think you’ll ever get it to fly with the filesystem watcher on the root. Syncthing will try to watch things in /proc, /dev/, whatnot which will likely not work well.
.