For example, I have a “Projects/” directory, which contains a whole lot of apps, I don’t want database.phps synced, as each laptop I work on has slightly different database.php settings. For this kind of share in particular, there are files that should be ignored by all nodes, so it makes sense for the ignore patterns themselves to be synced.
What I do - and seems to be the recommended approach in syncthing - is I manually create a .stignore file on each node that #include's .stignore_synced, which contains ignore patterns I want synced. So obviously I put a pattern like that on all nodes with that share. Thing is, every time I add a node, I have to remember to create the file. If you’re initializing a new node, the syncthing interface can create the folder, but any shared ignore patterns will be created after the folder is created. This means it is possible - and very likely - that your new node will end up downloading files you meant for it to ignore in the short span of time before you add the .stignore. Not a dealbreaker, but annoying.
Is there - or can there be - an equivalent of .stignore but synced between nodes by default? That way every share I define I can just write the .stignore_synced and be confident that every new share will always include those patterns upon initialization.
Would you consider adding a “synced by default” stignore file in addition to the current “private by default” stignore file?
I understand it might add complexity though, since by its nature a synced ignore file should be synced first and I don’t know of any priority mechanism within syncthing.
No. The current workaround is easy, and there is a huge ticket for brainstorming the next generation of ignore system - which most likely won’t use ignore files at all. Look for “next gen ignores” in the issue tracker.
Bumping because I hit this issue the first time I ever used syncthing, I’ve hit it each time I set up a new client, and I hit it again today.
For the record I’m happy to put time and / or money towards making this happen, it just seems silly to do either of those things if the response is going to be “we don’t want to make the current system work, we’re only interested in pull requests for the rewrite”, hence moaning here to gauge interest before I get out the text editor
The current workaround is easy
It’s easy when you remember to do it; even after using syncthing for years, I still forget to do it every time I add a new client to a folder It’s a UX anti-pattern – syncthing reporting “everything is configured and working” is only half the job, I need to go out of my way to ignore what syncthing is telling me, because there is still more work for me to do.
Look for “next gen ignores” in the issue tracker.
That lists “synced ignores” as a “maybe”, not a “requirement”
“What is the current state of this?”
“None, nobody is working on this.”[1]
Given that the next-gen ignores idea is 3 years old today (Happy birthday!) and seems to have made no progress in this area, could we re-consider implementing sync’ed ignores in the current system, instead of closing all the requests for it with “maybe it’ll happen as part of this distant-future rewrite which nobody is working on”?
[1] Except for adding a gui to the current .stignore file, which is excellent, but a completely separate issue
Synced ignore patterns quickly become difficult, think larger clusters where only some devices are online at the same time and the ignore patterns are changed in multiple places. We’d need to implement some sort of diff viewer and merge interface. (And we wouldn’t, because it goes against everything we stand for - so it’s just one point against.)
Personally, I’ve only rarely wanted synced ignores (as opposed to “the other devices don’t need to know these files exist”).
So all in all I wouldn’t hold my breath, given that it’s possible to accomplish it already, albeit with a separate step.
Sorry if I’m re-treading old ground here - but is there any mileage in having an ignores list that prevents a node from scanning certain local files?
This would allow you to suppress the broadcast of - for example - locally generated metadata, so you’d only have to add the ignore on a single node, rather than every other node in the cluster.
To me, there’s a slight mental disconnect between thinking “I’ve got this file on this machine that I don’t want synced” and “I need to go to every other machine and add that file to its Ignore List”…
@calmh, having had a fresh look at this request - plus the long issue tracker thread - after a year seems to bring a couple thoughts to the forefront.
a number of users seem to be agreeing with the “synced ignore” suggestion I made on that request on the grounds of “you’re bound to forget #include .stignore_synced on that one share”.
given that the current behavior of syncthing actually does unavoidably introduce the .stignore race condition already, I’m not really seeing how automating the .stignore_synced as an implicit #include .stignore_synced (what we’re recommending people manually do right now) line actually creates a problem. You don’t need a merge interface as that’s already what syncthing already does.
At any rate, though, I think such discussion should be moved to the next gen ignores thread instead. I hear ibizaman is having considerable progress on it.
I’m not disputing that people want synced ignores.
Syncthing currently does not merge anything, though.
I think it’s much more comfortable discussing here than in the GitHub issue tracker. Things in the tracker should be clear and actionable. I’m afraid the linked issue is somewhat beyond help already.
Right. Im not saying syncthing merges things, rather I’m saying we don’t need it to merge things.
Since the current advice unavoidably leads to the stignore race condition, and yet we still advise it anyways, then having a synced stignore (but seperate from the current one) can’t introduce problems we don’t already have, as all that would need to do is be an implicit #include .stignore_synced in the share.
Any conflicts or issues with the synced issue also exists in the current advice.
Nobody is working on synced ignores. Some work was done on the tree view in UI, but that’s about it. Also, given how long it took for us to settle on what’s the right approach, I wouldn’t be surprised if it never gets finished (atleast I would have abandoned it a long time ago).
Maybe we’re taking about different scenarios. When I think “synced ignore patterns” I no longer imagine patterns in a file on disk. I imagine them as stored somewhere in the database or config, and exchanged “out of band” (not as a file) in the protocol. This would then require some special handling for conflicts etc, as “conflict copies” etc would not be an option.
Perhaps you are just suggesting that .stignore should be synced like any other file. I’m not sure how to reconcile that with devices on the other side that don’t want synced patterns. Do I ignore .stignore? If the other side gives me an .stignore with .stignore in it, is it now ignored? How do I unignore it? All these could be decided on, but it’s not obviously non-confusing…
As per the original post (and also the next gen thread), the suggestion is for multiple levels of ignores.
.stignore stays as-is completely untouched by this proposal. (share+node level)
.stignore_synced is a separate file that has the exact same semantics as .stignore, but is synced between nodes (share level)
you can also have additional levels outside of these (e.g. config file level, which would be for the entire node), they are orthogonal to this
Right now, we recommend that you can get .stignore_synced behavior by creating (1) .stignore and within .stignore adding #include .stignore_synced or something along that line. Because at the moment, .stignore_synced is not treated specially, it will naturally be synced between all nodes in the share.
This recommendation that we are giving comes with all of the problems of making .stignore_synced an actual default feature of syncthing (without the need to #include it). Thus, the suggestion / feature cannot possibly introduce new problems because it itself is treated no differently than shared ignores right now.
Right, I agree. Adding one more pattern file that would also be synced would be quite easy I’m sure. Some details to sort out is precedence order between the files (probably non-synced should have higher precedence), and which file (or both) the GUI edits.
It won’t help you with your race condition at all though. Specifically, a device always does an initial scan of the folder at startup before talking to other devices. At that point there won’t have been a chance to sync down any ignore patterns.
Yeah, I suggested as much in the parenthesis. BTW I realized the race condition I’m talking about here may not be the same one you’re thinking of, as there are two of them.
Simultaneous edits to the ignore patterns on multiple devices. This problem remains however we do it, if using a synced ignore pattern file it’ll at least be swept under the rug by the usual conflict handling.
The fact that synced ignore patterns only take effect after the ignore patterns are actually synced. This is after everything has been scanned locally the first time, potentially much later (depending on how quick we are to sync and connect).
Number two may not be a feature killer, but it may be slightly surprising in some cases.
Wild idea on extending/updating the .stignore feature:
First: Use the file ‘.stignore’ (the current filename) and add to that the directory ‘.stignore.d’ and all files (recursive) in it.
Reason: There everyone can ad there and/or with dedicated ignore patterns for that area.
Second: Read the .stignore and .stignore.d (in this order) from:
the software installation location (default: file ‘.stignore’ containing “.stignore”)
the ‘standard map location’
the root of the current tree-to-sync.
Major change: do sync the .stignore file and .stignore.d tree in the current tree-to-sync. If you donnot want to do so, Just add “/.stignore” to one of your .stignore files.
To keep up with the current configuration: Add “/.stignore” to the .stignore in the installation directory. The I can put "!./stignore in the .stignore file in the three and it is synced to my wishes.
i suggested this in the Next Gen Ignores discussion, but it was apparently deemed off-topic, so i’ll add it here:
i think many people would benefit from ability to have just a simple
#(?!)include .stignores_sync type of flag that would allow syncthing to ignore a missing include file.
this is to address the race condition that occurs if you create a new share and expect the nodes receiving the share request to be created with the ignore rules including the shared ignore file(s).
a possible enhancement on the idea might be for folder syncs with a #(?!)include type rule in their .stignore to check for the file (first locally, then on the global index) to sync before any other files.
conflicts in synced ignore files would be treated like any other file conflict.
It’s a UX anti-pattern – syncthing reporting “everything is configured and working” is only half the job, I need to go out of my way to ignore what syncthing is telling me, because there is still more work for me to do.