From your list I can recognize only ST, borgbackup and rsync/rclone, the others are described too vaguely. Borgbackup is not for Android (there is an attempt), rsync is with limitations. All can be installed on Synology NAS. Borgbackup and rsync use SSH, ST is better in this regard. Also, Borgbackup and rsync are command-line tools, there’s no GUI (at least not for Synology). Etc.
@all But the most important point for me is that I have various kinds of folders (not only photos), and I do not want to use different tool for every type of folder. For instance, I was hesitating whether only to mirror a particular folder to NAS from my notebook, or whether to share this folder with other notebooks and use bi-directional synchronization. If I change my mind, why would I have to use another tool?
ST is advertised as “a continuous file synchronization program. It synchronizes files between two or more computers in real time, safely protected from prying eyes.”. I do not understand how one-way synchronization contradicts this motto. Wikipedia says (File synchronization - Wikipedia):
File synchronization (or syncing ) in computing is the process of ensuring that computer files in two or more locations are updated via certain rules.
In one-way file synchronization , also called mirroring, updated files are copied from a source location to one or more target locations, but no files are copied back to the source location. In two-way file synchronization , updated files are copied in both directions, usually with the purpose of keeping the two locations identical to each other. In this article, the term synchronization refers exclusively to two-way file synchronization.
So attempts to persuade us that one-way sync is not a sync are based on bad foundations.
It somehow hurts me every time I hear “ignore delete” and “backup” in one sentence. Ignore delete does not make for a backup. What happens if your phone decides to corrupt the photos? The “backup” (it’s not) gets corrupted. If you want to archive/backup your photos from the phone, the solution is dead simple: Use Syncthing to sync your data, including deleted, to your other device and do backups on that folder, using your favorite backup tool (mine right now is either rsync with some scripting or borgbackup, but restic and kopia look appealing to me too and there’s a myriad of other options).
What would this workflow be missing for you?
I think, no one claims that. Concerning photos, your suggestion wouldn’t work for me, because I actually do not use ST for backing up the photos but for incremental synchronization. Photos are not being taken just to be backed up, right? I do it this way:
Photos from the phones are incrementally synchronized (using ST) to NAS for a Photo Album app that we use for viewing the photos (Synology Photo Station).
A regular backup program (Synology Hyperbackup) backs up the photos with versioning from the Photo Station to somewhere else (to a backup disk).
Moreover, most backup programs do not have infinite history. If I followed your suggestion, how would I access the old already deleted photos?
A good backup program will allow you to expose backups as a read-only directory tree (e.g. borgbackup), in case you don’t need to write to it. Likely you do (metadata or something). Then an intermediate step is required: Periodically (or continuously, e.g. by FS watching) copy the content of the directory Syncthing syncs to another one (possibly using hardlinks to save space) - plain rsync would do, unison I think has watching inbuilt.
No, indeed nothing has infinite history. But yes of course every sensible backup program stores your data for as long as you tell it to (and provide the storage space).
Archive is probably the best term for what is happening here.
On a side note, when it comes to Android, there are plenty of popular 3rd party applications that do this kind of one-way “upload and forget” for services like Dropbox, OneDrive, Google Drive, MEGA, and many others. Some of those applications are even paid, and still have many users. I would argue that this clearly shows that people need this kind of functionality, regardless of the original meaning of “synchronisation” .
Photo Station requires to have data in a particular folder (/volume1/photo), nowhere else. I tried to store them elsewhere and mount them to the Photo Station tree, but this didn’t work.
Photo Station writes into the folders (thumbnails and metadata). But even if it did not, using the virtual filesystem of a backup program would be incredibly slow. Even now it is not very fast with the number of photos we have.
I’m not telling that scratching with the right hand behind the left ear is not possible, but who would want to do that?
Actually, I do not understand your (@AudriusButkevicius, @calmh, @imsodin and others) strong opposition against one-way sync and against ignoring the deletes. Tell me a good reason why ST should not support these features. @AudriusButkevicius mentioned that it is now implemented as a dirty hack - is this the only reason??
There’s no opposition to one way sync. We have lots of features to support one way sync - and one feature to ignore deletes. The latter happens to be implemented in a crappy way and should be improved. It also goes a bit against the grain by being a feature to specifically prevent syncing some operations. That’s somewhat tricky when the rest of the system works hard to make sure to sync all operations.
Main problem: It’s not clear what we are talking about here respectively lots of things are mixed. Ignore delete is not at all the same thing nor strongly linked with one-way sync.
Then there’s one reason: I am not interested in it, neither is (I am quite sure) Audrius. That’s not a logical/technical argument around a problem, that’s a factual statement that can’t be argued. We just won’t work on it. I might be motivated to replace ignore deletes to get rid of an ugly hack, but not because I want the replacement.
And when Syncthing was written and still now, it was meant to be continuous. decentralized synchronisation. Lots of features got added along the way, but at it’s core it’s still that. Meaning the internal architecture, the protocol, … all is wired to that purpose. Features either need to align well with that, or be a non-breaking extension where the benefit outweighs the cost (of writing and maintaining it).
Is that so different from .stignore files? In bsync I mentioned above it is not, it’s just a dynamic filtration rule. I do not know how about ST.
I think, we’ve already participated in a similar discussion concerning directory junctions. There’s a difference if you claim that you’re strongly against some feature (e.g. because you feel it does not belong to this kind of application), and that you do not want to implement it yourself.
In the current implementation it’s very different from ignore patterns. One could imagine otherwise. Even a hack to react on deleted files and add the them to ignore patterns would be better (but probably not scale, etc.)
I think one major difference between the typical one-way propagation and Syncthing’s decentralized synchronization is the number of devices people are thinking about. With any number greater than two of equal importance (think of cluster anarchy), who decides what the one way is?
Syncthing solves the ambiguity between many participants by deducing one common, global target state, which can not be declared solely by one or two devices. That’s why the send or receive only folder types always just concern the local device behavior for itself.
I’d agree that the best solution is to do one-way propagation locally from a regular all-way synchronized Syncthing folder. Possibly protecting whatever device sees itself as an authority by making that send-only.
@acolomb I mentioned bsync on purpose because it is not a “typical one-way propagation” tool. It is build on top of rsync and allows mutual synchronization or one-way mirroring of several folders (participants). In many aspects it does the same as ST, but of course, there are also many differences. But the conceptual problems that are solved in bsync are similar to problems in ST.
However, it still appears to me that the problems of Ignore Delete are related to problems of one-way-sync, as they manifest themselves similarly. For instance:
For bi-directional sync + Ignore Deletes at target, if I delete files at source, the deletes are marked as out-of-sync at the source device. When I change the sync to uni-directional, the out-of-sync deletes do not disappear, and I do not know a way how to get rid of them.
Similarly, when the source folder is Send Only but target is Send-Receive, then changes at the target side cause the source folder unsynchronized. This state is unresolvable, even after changing target to Receive Only.
For uni-directional sync, when target was modified locally, the target device is stuck in the “Syncing 95%” state with plenty of out-of-sync items. I do not know how to resolve this situation.
When the target folder is Receive Only, it is still in the state “Local Additions” with plenty of Locally Changed items.
I wanted to create a new thread for these issues. Here I want only to point out that all these issues are very similar despite that some of them are problems of Ignore Deletes and other of uni-directional sync.
The quirks in bad state after changing folder types are genuine bugs and I think we should fix as part of the offering.
Ignore deletes is a hack, and until some superhero steps up to solve it, I think we should discourage people from using it and aim to deprecate it.
What you are talking about with ignore deletes is not actually syncing, its just transfers from A to B (given that you want to delete stuff and not want that to sync back). It doesn’t fit with the tool and its goals, but I have no objections of having it as a feature if the super hero comes and fixes it.
Until then, don’t use it, the most likely outcome given none of the maintainers wants to work on this, is that we’ll pull it sooner or later.
What problems in particular does Ignore Deletes cause? I’ve noted no problems apart of “cosmetic” problems I mentioned above. Based on the discussion, the main problem seems to me that if someone breaks it, then no one will want to fix it.
How this corruption manifests itself? Does it cause synchronization errors or what?
And how often does this corruption happen? Every time a sync is performed, or only occasionally?
It’s forever in a bad unrecoverable state, where everyone sees the device out of sync with no way of fixing it, even if you disable ignore deletes, and it doesn’t do the right thing (delete files) if you disable it.
So does it have the symptoms I described in points 1.-4. above (at 23:49)? This seems an annoying but rather “cosmetic” problem to me. Does it work correctly when I leave Ignore Deletes checked on (and don’t disable it)?
Apologies if this would belong better elsewhere. I came to this thread thinking I wanted this feature for photo backups, but after reading the thread and thinking about it a bit more, what I actually want is a way to limit which files are stored locally for some space-constrained nodes.
Photos is indeed one example, where I want a full copy on my NAS and my desktop, with edits and explicit deletions synchronized, but also want a way to limit what photos are stored on my phone (e.g., by purging local copies of old photos) to save storage space.
Documents is another example. I want full copies on my Desktop, Laptop, and NAS, but my Chrome OS device has much less storage, so I’d like to be able only to store the most recently modified documents up to a max of, e.g., 1GiB. (An option to keep documents modified in the last n months would probably also be workable.) Additionally, I’d like the ability to browse / filter the index and add explicit overrides to, e.g., avoid maintaining a large file locally that isn’t useful on the device, or, alternatively, to request an older file be synced that wouldn’t be otherwise.
In any case, I’d like a safeguard that a file wouldn’t be removed locally until it was confirmed to be synced to a connected device.
.