Selective Sync Feature(most important)

Hello, I switched to Resilio some time ago due to this feature missing, and could be interested in contributing some code to bring this to Syncthing. But as I’m not very involved in the community, could you (or another maintainer) point me at previous (stalled) attempts at designing or implementing things in this space? So I don’t duplicate any work, and have something to use for inspiration. Thanks!

5 Likes

From a quick GitHub search, there might be more:

Could also find some PR’s linked here:

1 Like

I’m thinking that a tree view is not the only piece of the puzzle, something needs to be done about like renaming or creating folders at the root. If you do this with a ‘*’ ignore at the end, it’s silently not synced… The correct (to me) behavior seems to not really be achievable with the current ignore feature. If a new folder is created on another node, don’t sync it to me. If I create a new folder, start syncing it.

I think ignores are smart enough to exclude things along the way of the item being excluded.

I’m contemplating of adding such a feature to Syncthing Tray. So far I added a UI for the browse-API. It would be nice if one could select files/directories from there and ignore/unignore them easily. An “ignore and remove locally” feature might also be useful. However, I’ll have too see how far I can come with ignore patterns within the time I can spend on it.

Of course it would also be more beneficial for everyone if such a feature was implemented in Syncthing’s official GUI but that’s probably harder to do.

(I’m also contemplating of porting Syncthing Tray to Android where this feature would be most beneficial to have because one the phone editing ignore patterns is most inconvenient. However, the Android/App port is probably a lot of work - although probably also a quite interesting piece of work.)

2 Likes

I’m kind of curious the percentage of Syncthing users that use the tray UIs…

I would like to have a documents / music files on a NAS and be able to sync (make available offline) some files when I go on vacation / travel.

2 Likes

Wouldn’t just adding an option to ignore all files when starting a new sync allow it to be managed by an external program through the .stignore ?

1 Like

This feature needs more votes in the Syncthing Roadmap Voting

3 Likes

So in my “bubble” my wife understood almost none of this. I on the other hand . . I understood about 80-85%. Which means in my “bubble” I am tech savvy. But I couldn’t have written one single reply as intelligently (or correctly) as single one of you. You’re all light years ahead of me. Ya maybe I smoked before writing this. But damn. Ya’ll nerds are impressive af. Bravo with your syncing.

It’s really frustrating that I have to install Resilio Sync on my new phone because because this still isn’t possible with Syncthing. :confounded: (And all the GitHub issues about it are locked so we can’t even vote on them.)

2 Likes

I use ignore patterns for my phone to only sync my kdbx file on the phone.

There is now an iOS app in beta that supports selective synchronization. It basically uses the trick mentioned a few times before in this thread, which is to ‘ignore’ all paths except those explicitly synchronized. There are indeed plenty of edge cases with this approach (and the app deals with those in specific ways, in part due to limitations of iOS, and this may or may not be desirable in all your use cases). Notably, renames are handled as delete+create (which means the file is deleted from peers, then only synced when the user accepts the ‘extra file’ in the app, which will detect it and warn about it).

Interested developers should look at folder.go in the app source code (look for methods concerning ‘selected files’, ‘selective folders’ and ‘extraneous files’).

Note that in addition to selective synchronization, the app provides on-demand (read only) access to files not stored locally (when you open a file it downloads or even streams it from another peer). This perhaps fulfills other scenarios for which selective sync is desired. (For this feature, we simply ‘ignore’ everything so nothing is synced for a folder, except for the ‘global file tree’. The feature can work in conjunction with selective sync as well - the on-demand feature works for all files that are not locally synced).

2 Likes

I get that entitlement is a problem with open-source projects. I’m not demanding, but I do want to add to the chorus that this is an extremely key feature for me and what’s holding me back from using Syncthing. The utility value is halved lacking it, so it’s a total mystery to me why the maintainers don’t find it to be sufficiently interesting.

In an increasingly mobile, multi-device world, Syncthing could be a massive boon for data sovereignty and general organization and productivity for billions of people with this feature. They’ll always retain some share due to marketing, but the actual necessary use-cases for corporate cloud storage solutions like Drive and Dropbox would be vanishingly small. I have a hard time groking why providing something that important wouldn’t be a priority.

17,000 people cared about this issue enough to Google it and read through a forum thread, so I think it’s pretty clear that the interest is there.

Upthread, it’s been explained repeatedly that the current maintainers of this open source project aren’t interested in working on it.

If it’s important to you, I would respectfully suggest that you develop the feature yourself. I don’t have that skillset, so if it were that important to me, I would hire a developer to build it, or lead an effort to fund it.

And enough interest from the community of users/potential users might change their minds. If not, that’s their prerogative, and I think I was clear that I understand that. But open-source works better when end-users participate and are clear about what’s important to them and what’s not working.

I don’t have the skillset, unfortunately, nor the funds to hire anyone. I would be happy to contribute to a crowdfunding effort. As for leading the effort, that probably makes more sense for someone who better understands at the least the technical outline for implementation. I, frankly, don’t–I’m only a minorly nerdy ordinary person considering whether to get Resilio up and running on my machines again or switch to Syncthing.

My reply below isn’t intended to criticize and/or nitpick your reasons for advocating that Syncthing support selective sync, so please don’t take it as such.

It’s only to share a perspective as someone on the “other side of the fence” who used to do more coding before shifting to a sysadmin role supporting software development teams.

A lot, if not most, open-source projects that aren’t seeking a for-profit business model are born out of necessity – the author(s) needed a tool that wasn’t available or the existing ones didn’t check all of their boxes, so they started from a fork or from scratch. Then if the rest of us are fortunate, we get the benefit of their hard work for free.

I’m assuming that you were just exaggerating… :smile:

From a technical perspective, I’m not sure the current Syncthing discovery and relay infrastructure would be able to handle tens of millions of devices, let alone billions.

Most of those new users are unlikely to have the interest and/or skill to set up and maintain their own discovery and/or relay servers, so somehow the costs of maintaining those servers would have to be covered.

I’m not so sure about that…

From what I’ve seen on this forum, minimizing the cost of storage generally seems to rank higher in importance than data sovereignty and other features. If there were service providers that offered lots of free storage (much more than the typical 5 or 10 GB), that’s where most users would likely be for many obvious reasons.

(Side note: Last time I checked, ~4% of Dropbox’s user base are paying customers. With >700 million users, it’s roughly 1.2 million terabytes of storage for the 2GB free basic tier users. Net income was negative in 2023.)

While commercial cloud storage is often used for convenient access to frequently used files across multiple devices, selective sync isn’t as commonly used as some assume. When given the chance, most people sync everything (as a form of “backup” and for convenience), or do sync-on-demand (which is different from the selective sync people have been asking for in Syncthing – Google, Dropbox and others refer to it as something like “make available offline”).

(My employer subscribes to multiple cloud storage service providers with tens of thousands of users. Although fully capable of providing it in-house, the infrastructure and support costs outweighed the subscription costs.)

I get it, but consider who it’s a priority for – if and when it ever becomes important enough to one of the Syncthing developers, it’ll happen.

Although money can provide some motivation, software development is part art and part science, so being actually interested in working on a selective sync feature plays a huge part (e.g. it’s unlikely there are developers working for a video game company that have no interest in video games).

If all 17,000 were to contribute $5 each to a fund, there might be enough to hire a developer to write one and maintain it… for a year. :smirk:

Kidding aside, Android and iOS are moving targets with major releases every year like clockwork for the past 10+ years. So developing a selective sync feature isn’t going to be a one-and-done project. The code will have to be maintained at least on an occasional basis – and almost guaranteed – no combination of features and/or UI design will satisfy everyone so there’s going to be a lot of work fielding support requests.

According to the available telemetry data, 23.4% of Syncthing devices are running Android. Since the odds are that the majority of the 17,000 people who Googled “selective sync” are interested because they don’t have unlimited storage and/or bandwidth on their mobile devices, it’s reasonable to conclude that support for Syncthing on Android would be an important requirement.

3 Likes

Fair responses, thank you. I guess I’m just upset that the only way I can manage smooth transition between devices remotely efficiently is by trusting all my files to a closed-source resource-hog that isn’t very actively maintained. Something like this just seems like really important 21st century infrastructure to me. Google et al., beyond being a privacy nightmare and a monthly expense (for anything serious), is also a PITA way to manage things that only barely improves on physical media for non-social use.

I do take your points. I just wish an alternative existed.

I only skimmed over the new posts, so please excuse me if I don’t perfectly represent some parts or miss others. I just have an urge to comment on the “maintainers this, maintainers that” parts:

I believe maintainers do consider this important, and I know at least one does (I). However that doesn’t really come into it, whether or not something like selective sync happens or not doesn’t depend on what maintainers think of this (beyond that basic “yes, the feature is welcome” stance that you’ll find all over already for this). We are all full-time employed (not to work on syncthing) and we need to deal with all things syncthing (that’s what maintainer means). There’s simply not much time left for us to “do things”, especially not big changes. And besides the churn work, there are also a crapton of “very important” things, this is just one of them. And yes, we tend to prioritise issues both by our goals and our interests/abilities, where the latter admittedly doesn’t lie on the front-end for any of us. So it’s accurate to say that you shouldn’t hold your breath for us to implement this. However there’s already one third-party tool that does it, one beta ios app that does some of it, one alpha quality abandoned attempt at an android app (and java/kotlin BEP implementation), multiple PRs attempting to add it to our UI (remember at least 2, one of which quite recent), … What I am saying is that syncthing and the ignore system functions in a way, that all the pieces are already there for some kind of selective sync feature, and we maintainers will definitely not stay in the way of it, the contrary.

Oh wow, typically for me I just wanted to write something short about “maintainers”, and end up with a wall of text.

tl;dr:
Don’t focus on maintainers. We will not stand in the way of getting this feature done (the opposite), syncthing already has pieces in place to make some variant of it possible and we’ll keep working to make sure syncthing stays usable (and thus ready for this feature) - indeed for a variety of reasons it’s unlikely that we’ll implement the feature.

(Damn, even my tl;dr is a classic German-style bandwagon of a sentence (well two). :smiley: )

8 Likes

Hi,

The original poster at the very top wanted to be able to share with other people some of the folders and files that are in his Syncthing folders, but not all.

I tested a solution that works for me and that is to create a copy of the data on a cloud drive. In my case I’m using Pcloud.

Now that I have the data on a Pcloud drive, I simply use the Pcloud file and folder sharing feature to share the data that I need to with others bypassing the need to have syncthing do the work itself.

I know that people who are using this software are in some cases doing so to avoid having to use a cloud drive however in this case it is to me a logical solution to the original poster’s problem.