Selective Sync Feature(most important)

When I really can’t do it directly on the local device I do it remotely (ssh or Remote desktop over ssh). Whatever I admit tweaking an external text file is boring, even it’s local, and that’s what I do when I need symmetric ignore or remote ignore management (via #include files). For local only management (e.g. huge remote Pictures Folder from which I just want to locally work on files from a subtree) I do with ignore patterns directly in the local GUI, what I admit is just a little less boring than with external file… ok, sorry for the noise.

Not noise. All part of the discussion.

I have a similar use case. A 30TB NAS with a ton of stuff and I’d like to selectively sync stuff on a laptop and a phone.

Currently only a small subset is under syncthing management because I can’t selectively sync. (Who has 30TB of storage on a laptop?)

Anyway im not at all familiar with the syncthing codebase but considering taking a look at what it would take to be able to implement something here.

4 Likes

Hello, I switched to Resilio some time ago due to this feature missing, and could be interested in contributing some code to bring this to Syncthing. But as I’m not very involved in the community, could you (or another maintainer) point me at previous (stalled) attempts at designing or implementing things in this space? So I don’t duplicate any work, and have something to use for inspiration. Thanks!

5 Likes

From a quick GitHub search, there might be more:

Could also find some PR’s linked here:

1 Like

I’m thinking that a tree view is not the only piece of the puzzle, something needs to be done about like renaming or creating folders at the root. If you do this with a ‘*’ ignore at the end, it’s silently not synced… The correct (to me) behavior seems to not really be achievable with the current ignore feature. If a new folder is created on another node, don’t sync it to me. If I create a new folder, start syncing it.

I think ignores are smart enough to exclude things along the way of the item being excluded.

I’m contemplating of adding such a feature to Syncthing Tray. So far I added a UI for the browse-API. It would be nice if one could select files/directories from there and ignore/unignore them easily. An “ignore and remove locally” feature might also be useful. However, I’ll have too see how far I can come with ignore patterns within the time I can spend on it.

Of course it would also be more beneficial for everyone if such a feature was implemented in Syncthing’s official GUI but that’s probably harder to do.

(I’m also contemplating of porting Syncthing Tray to Android where this feature would be most beneficial to have because one the phone editing ignore patterns is most inconvenient. However, the Android/App port is probably a lot of work - although probably also a quite interesting piece of work.)

2 Likes

I’m kind of curious the percentage of Syncthing users that use the tray UIs…

I would like to have a documents / music files on a NAS and be able to sync (make available offline) some files when I go on vacation / travel.

2 Likes

Wouldn’t just adding an option to ignore all files when starting a new sync allow it to be managed by an external program through the .stignore ?

1 Like

This feature needs more votes in the Syncthing Roadmap Voting

3 Likes

So in my “bubble” my wife understood almost none of this. I on the other hand . . I understood about 80-85%. Which means in my “bubble” I am tech savvy. But I couldn’t have written one single reply as intelligently (or correctly) as single one of you. You’re all light years ahead of me. Ya maybe I smoked before writing this. But damn. Ya’ll nerds are impressive af. Bravo with your syncing.

It’s really frustrating that I have to install Resilio Sync on my new phone because because this still isn’t possible with Syncthing. :confounded: (And all the GitHub issues about it are locked so we can’t even vote on them.)

2 Likes

I use ignore patterns for my phone to only sync my kdbx file on the phone.

There is now an iOS app in beta that supports selective synchronization. It basically uses the trick mentioned a few times before in this thread, which is to ‘ignore’ all paths except those explicitly synchronized. There are indeed plenty of edge cases with this approach (and the app deals with those in specific ways, in part due to limitations of iOS, and this may or may not be desirable in all your use cases). Notably, renames are handled as delete+create (which means the file is deleted from peers, then only synced when the user accepts the ‘extra file’ in the app, which will detect it and warn about it).

Interested developers should look at folder.go in the app source code (look for methods concerning ‘selected files’, ‘selective folders’ and ‘extraneous files’).

Note that in addition to selective synchronization, the app provides on-demand (read only) access to files not stored locally (when you open a file it downloads or even streams it from another peer). This perhaps fulfills other scenarios for which selective sync is desired. (For this feature, we simply ‘ignore’ everything so nothing is synced for a folder, except for the ‘global file tree’. The feature can work in conjunction with selective sync as well - the on-demand feature works for all files that are not locally synced).

2 Likes

I get that entitlement is a problem with open-source projects. I’m not demanding, but I do want to add to the chorus that this is an extremely key feature for me and what’s holding me back from using Syncthing. The utility value is halved lacking it, so it’s a total mystery to me why the maintainers don’t find it to be sufficiently interesting.

In an increasingly mobile, multi-device world, Syncthing could be a massive boon for data sovereignty and general organization and productivity for billions of people with this feature. They’ll always retain some share due to marketing, but the actual necessary use-cases for corporate cloud storage solutions like Drive and Dropbox would be vanishingly small. I have a hard time groking why providing something that important wouldn’t be a priority.

17,000 people cared about this issue enough to Google it and read through a forum thread, so I think it’s pretty clear that the interest is there.

Upthread, it’s been explained repeatedly that the current maintainers of this open source project aren’t interested in working on it.

If it’s important to you, I would respectfully suggest that you develop the feature yourself. I don’t have that skillset, so if it were that important to me, I would hire a developer to build it, or lead an effort to fund it.

And enough interest from the community of users/potential users might change their minds. If not, that’s their prerogative, and I think I was clear that I understand that. But open-source works better when end-users participate and are clear about what’s important to them and what’s not working.

I don’t have the skillset, unfortunately, nor the funds to hire anyone. I would be happy to contribute to a crowdfunding effort. As for leading the effort, that probably makes more sense for someone who better understands at the least the technical outline for implementation. I, frankly, don’t–I’m only a minorly nerdy ordinary person considering whether to get Resilio up and running on my machines again or switch to Syncthing.

My reply below isn’t intended to criticize and/or nitpick your reasons for advocating that Syncthing support selective sync, so please don’t take it as such.

It’s only to share a perspective as someone on the “other side of the fence” who used to do more coding before shifting to a sysadmin role supporting software development teams.

A lot, if not most, open-source projects that aren’t seeking a for-profit business model are born out of necessity – the author(s) needed a tool that wasn’t available or the existing ones didn’t check all of their boxes, so they started from a fork or from scratch. Then if the rest of us are fortunate, we get the benefit of their hard work for free.

I’m assuming that you were just exaggerating… :smile:

From a technical perspective, I’m not sure the current Syncthing discovery and relay infrastructure would be able to handle tens of millions of devices, let alone billions.

Most of those new users are unlikely to have the interest and/or skill to set up and maintain their own discovery and/or relay servers, so somehow the costs of maintaining those servers would have to be covered.

I’m not so sure about that…

From what I’ve seen on this forum, minimizing the cost of storage generally seems to rank higher in importance than data sovereignty and other features. If there were service providers that offered lots of free storage (much more than the typical 5 or 10 GB), that’s where most users would likely be for many obvious reasons.

(Side note: Last time I checked, ~4% of Dropbox’s user base are paying customers. With >700 million users, it’s roughly 1.2 million terabytes of storage for the 2GB free basic tier users. Net income was negative in 2023.)

While commercial cloud storage is often used for convenient access to frequently used files across multiple devices, selective sync isn’t as commonly used as some assume. When given the chance, most people sync everything (as a form of “backup” and for convenience), or do sync-on-demand (which is different from the selective sync people have been asking for in Syncthing – Google, Dropbox and others refer to it as something like “make available offline”).

(My employer subscribes to multiple cloud storage service providers with tens of thousands of users. Although fully capable of providing it in-house, the infrastructure and support costs outweighed the subscription costs.)

I get it, but consider who it’s a priority for – if and when it ever becomes important enough to one of the Syncthing developers, it’ll happen.

Although money can provide some motivation, software development is part art and part science, so being actually interested in working on a selective sync feature plays a huge part (e.g. it’s unlikely there are developers working for a video game company that have no interest in video games).

If all 17,000 were to contribute $5 each to a fund, there might be enough to hire a developer to write one and maintain it… for a year. :smirk:

Kidding aside, Android and iOS are moving targets with major releases every year like clockwork for the past 10+ years. So developing a selective sync feature isn’t going to be a one-and-done project. The code will have to be maintained at least on an occasional basis – and almost guaranteed – no combination of features and/or UI design will satisfy everyone so there’s going to be a lot of work fielding support requests.

According to the available telemetry data, 23.4% of Syncthing devices are running Android. Since the odds are that the majority of the 17,000 people who Googled “selective sync” are interested because they don’t have unlimited storage and/or bandwidth on their mobile devices, it’s reasonable to conclude that support for Syncthing on Android would be an important requirement.

3 Likes

Fair responses, thank you. I guess I’m just upset that the only way I can manage smooth transition between devices remotely efficiently is by trusting all my files to a closed-source resource-hog that isn’t very actively maintained. Something like this just seems like really important 21st century infrastructure to me. Google et al., beyond being a privacy nightmare and a monthly expense (for anything serious), is also a PITA way to manage things that only barely improves on physical media for non-social use.

I do take your points. I just wish an alternative existed.