This proposal is almost certainly not fully baked. Looking for thoughts, especially pointers to previous discussion if something like this has been proposed and shot down before.
Motivation
.stignore can be used to approximate the “selective sync” feature of other file syncing programs, like this
*
!/projects/foo
!/documents
But there’s one catch: if I create a new directory in an ignored path, perhaps /projects/newthing, it will silently not get synced, contrary to my expectations from e.g. Resilio Sync, which will still upload anything created locally in selective sync mode.
Proposal
Add a new special file to use instead of .stignore for this use case, called .strequest. [Consider the name to be a placeholder :P]
A path included (and not excluded) by .strequest is considered “requested”. This defaults to requested, i.e. if the file is empty, everything is requested. [Possibly confusingly different from .stignore - an empty .stignore means “nothing”, while an empty .strequest means “everything”. But it makes more sense when you think of “ignore nothing” and “request everything” as being the same.]
Semantics during syncing:
If an item is created/modified/deleted either locally or remotely, and the path is in .stignore, nothing happens. (Nothing new here.)
If an item is created/modified/deleted locally, the change is uploaded to peers, regardless of whether the path is requested.
If an item is modified/deleted remotely, and the path exists locally, the change is downloaded, again regardless of whether the path is requested.
If an item is created remotely, and the path does not exist locally, it will be downloaded only if the path is requested. (<-- main difference!)
I haven’t used Dropbox in years, could someone familiar with it comment on whether this conforms to user expectations in relation to its selective sync implementation?
Dropbox’s selective sync is actually more like Syncthing’s existing .stignore with different starting patterns depending on if the host is a mobile device or not.
On Android, Dropbox behaves as if Syncthing has a .stignore with the following pattern:
*
The Dropbox mobile app then presents a virtual tree of the files and folders in a Dropbox account with the option to selectively sync – or more accurately, un-ignore – individual files and/or folders that have been marked. The selected items are downloaded and stored in Dropbox’s local cache, then refreshed whenever changes are detected.
For the desktop Dropbox client, it’s as if Syncthing starts with an empty .stignore, where everything is synced by default. A user can then selectively mark files and folders (i.e. add patterns to .stignore) to remove them from the local cache.
So Syncthing’s existing .stignore already supports the necessary pieces for selective sync. Start with *, then add/remove patterns starting with ! as needed.
Interesting, I suppose what I’m hoping to achieve is closer to the mobile client behavior. But what happens if you create a new folder on Android dropbox, is it automatically added to the selective sync?
Realized a problem: given this proposal it’s easy to select a folder to be downloaded, but unselecting it is impossible - if you remove it from .strequest local changes are still synced, so deleting it will delete everywhere.
There is a similar, though less serious, problem with using .stignore for this - if you want to stop having a subdirectory locally and add an ignore pattern for it, you then have to go and manually delete it.
Compare to resilio sync, where you right click the directory and choose “remove from this device”.
Don’t push deletes for non-requested files. The files just stay on other peers forever. Probably a bad idea, seems like a confusing state to be in. Closer to resilio sync behavior, the difference being that resilio creates placeholder files to show that the file is still on other peers, and you can delete the placeholder to actually delete the file.
Some system of automatically adding locally created files to .strequest…? Closer to how dropbox works, I think.
Part of the confusion that many who want Syncthing to be like Dropbox have is that Dropbox is a hub-and-spoke topology. Its offline/selective sync feature is about what is to be downloaded from the cloud to be locally cached instead of what’s to be synced to other peers. Each peer individually decides what it wants from cloud storage.
Syncthing’s core feature is peer-to-peer bi-directional sync – mirroring between peers without a hub/master node.
I don’t see why this matters - the cloud is an implementation detail and should not be relevant for the UX. Replace “the cloud” with “other peers” and everything makes sense for syncthing. For example
Each syncthing peer individually decides what it wants from the other peers.
Where I think you’re right is that syncthing currently doesn’t make a distinction between choosing what to pull (in dropbox, selective sync) and choosing what to push (ignores). The point of the proposal is to highlight a place where it makes sense to treat these a bit differently.
CurlFtpFS is a FTP client that connects to a FTP server and downloads files on-demand, optionally caching files locally. From the user’s perspective, they see a virtual directory tree to select files/folders from. New files/folders created locally are pushed to a FTP server.
CurlFtpFS vs. Dropbox…
Dropbox app is a Dropbox client that connects to a Dropbox server and downloads files on-demand, optionally caching files locally. From the user’s perspective, they see a virtual directory tree to select files/folders from. New files/folders created locally are pushed to a Dropox server.
CurlFtpFS and Dropbox have much more in common than with Syncthing. Some on this forum have combined Syncthing with Samba, WebDAV, and/or some other file server software.
You’re right with regards to it being a UX design solution. There are several projects that help manage Syncthing’s .stignore file along with multiple forum threads about revamping Syncthing’s web UI to support a similar feature, but it’s one of those things that conceptually sounds simple, but is complicated to implement (a lot of things in end-user software development tend to be that way). And with the official Android app in maintenance mode – and on it’s way out of Google Play by the end of the year – it’s not likely to happen anytime soon.
“The Cloud” is not just an implementation detail, it is also a company that specializes in storing data in a geographically redundant fashion.
Imagine you have a shared folder for projects, people request/synchronize the projects they’re working on, don’t synchronize the files they don’t personally need. Then the only person with Project 4 on their drive formats and re-installs their OS because everything will just sync up like Dropbox.
You can argue that the underlying design was problematic, and I’ll agree, but it is important to design a UI with the understanding of the fundamental difference: The cloud is not “just somebody else’s computer”
For Syncthing specifically: Why? Please help me understand what your use case would be for Syncthing where the UI has to understand that any given device was provided by a cloud vendor?
The background: With cloud-based systems, the selective synchronization feature allows you to select what data is synchronized to your computer while all data remains on the cloud and available forever (within the terms of your contract).
This is not the case with a theoretical “Selective Synchronization”, should it be added to Syncthing, were the exact feature implemented in the same way.
We are talking about a fundamentally different implementation of a similar-looking feature, and if users treat it the same way as they treat Dropbox’s selective sync they will lose data.
Technically, this situation is already possible with ignore files, and if you look closely you’ll notice it has an entirely different UI when compared to a Dropbox-style “list of available folders, pick which ones you’d like to have available locally, and please have no hesitation to add or remove them from the list, your data will be fine”.
FWIW: in the Synctrain iOS app, a folder can either be added as ‘fully synced’ or ‘selectively synced’. In the latter case, the assumption is that some other device has a full copy (and will keep it, i.e. is ‘the cloud’ you refer to) and you can indeed select and deselect files for local storage and syncing (everything else is ignored but is additionally available on-demand, you can actually stream i.e. video from other peers!). This makes sense on mobile (imo) because it is primarily a consumption device and the assumption is that the mobile device can get lost at any time (with all its data).
I agree that for a full-blown peer, the base assumption is not just consumption but also actual storage (your ‘cloud’ role) and hence selective sync could be trickier for users who do not know what they are doing.
That said, the ‘selective sync’ model is very useful in a scenario with a central NAS or two that have full copies, and a few mobile devices that want to access only a subset and are not required to take part in storage (may lose data at any time). This scenario is not peer-to-peer (at least not equal peers!) but rather client-to-peers if you will and I agree this is a fundamental difference with Dropbox (which is client-server).