Selective Sync Feature(most important)

Feature
#1

Hey there, Syncthing Team, thanks for providing this wonderful app.

this app is wonderful but all the time i keep feeling the need of a feature -

Selective sync

this feature should be your top priority because everything is going fine. Rsilio Sync has this feature but that app is slow in working and not constant like syncthing. Sometimes i Need to share my hard drive data through syncthing. I don’t want all the data , i need different files from different different folder. So now if i share all the drive at once it’s gonna download all data and thus it takes full space. So please provide SELECTIVE SYNC FEATURE.

2 Likes
#2

Syncthing-Lite :slight_smile:

#3

Except that was in alpha state and isn’t maintained/is archived for quite a while now.

Try to not use that word, enhances your chances of getting meaningful and sympathetic reactions :wink:

You don’t explain why ignore patterns don’t work for you: Ignoring Files — Syncthing v1 documentation
I guess too complicated/no GUI interface. Agreed, having that would be nice.

#4

okay,

if you feel “should” word was meant to disrespect you then I’m taking back my word Bro.

Yeah it’s complicated but i can handle that. I never tried “ignoring files” feature. I think it means syncthing will not sync those files from that particular folder or is it (ignoring files) means the same as selective sync like , we can see the name of the file in our list and if we want it we can click on it to download it??? also bro selective sync means , as far as i understand is you'll get all the list of files in that folder but not synced so they are not taking any space in hard drive but if you want any file you can click on it to download it.

#5

Click to download file sounds like a web server, not an application that syncs files. As there is no syncing, its just “click to download”, I am not sure that feature fits in well with syncthing.

#6

This what Dropbox calls “smart sync”, i.e. the files appear in the system, but take only 0 bytes, and are only downloaded once actually used. From then on, they just become local files. It is “cool”, but probably very difficult to implement on a file system level in all the different operating systems.

#7

With selective sync, Resilio first creates a complete directory together with the files of the folder with the help of links. Then in a little browser it is chosen what should be physically synced. The selected folders and files will be downloaded and synced normally. If you want to have additional folders or individual files physically on your computer, the links are processed accordingly. So far for the function there.

Something similar would have to be built into Syncthing as a feature. For now, I’d just work with Ignore Patterns. That would mean opening the list, editing it accordingly and closing it again. I think that is reasonable. This becomes easier if complete folders are ignored, as otherwise an overly long Ignore List can arise, which could also negatively affect performance. However, it is then not possible to see directly what physical elements are still available remotely and which could be downloaded. Such elements are seen in the ignore list.

It looks different when files are added, deleted or renamed, etc. In everything, the ignore list must be in the foreground. That can sometimes go wrong if it is forgotten.

#8

Something like this will be great, this is the only thing I can think of to improve Syncthing.

2 Likes
#9

This is the main reason I went back from Syncthing to Resilio Sync… The selective sync feature allows me to have a massive file share, like my photos or video library on my NAS and have it synced with a laptop, but only show what is there. If I need a file, I double click and it downloads and is usable… Ideally, it would use (for Windows anyway, and I think Mac has a similar API) the Cloud Filter API (Build a Cloud Sync Engine that Supports Placeholder Files - Win32 apps | Microsoft Docs) that allows all files to be shown on the machine, but only accessible when double clicked or opened in an editor/other program. Apple has a similar one, I just can’t find the docs, and not sure about Linux… Both Dropbox and One Drive use both Apple and Microsoft’s implementation in their apps (IIRC, Dropbox uses the Microsoft API… might be wrong there).

#10

I’ve used Resilio for years and have already worked with the former BTsync. There’s no question about it, with local sync, selective sync, folder sharing via identity, etc., Resilio is clearly a tool for fans of features. And well, Resilio still runs on most platforms, but unfortunately not as flawlessly as they have progressed.

Of course, I would like all these features, which are obvious for everyday life, also with Syncthing. However, I turned away from Resilio for two main reasons. The consumption of resources, which becomes a problem with small network bandwidths and especially because of the update policy.

At Syncthing we have the advantage that the software is optimized and further developed in small and clear steps. Up until a few years ago, this was similar with Resilio, but nothing seems to be happening now. This worries many users and that is one of the reasons why quite a few are turning away from Resilio and continue to work with Syncthing, because the maintainers take care of the users and their problems and the software is continuously being further developed and adapted to the new operating systems.

And it is precisely these advantages that you no longer have with Resilio, at least for the moment. So whether the software is open source or not is no longer important as a reason for switching.

#11

Maybe an croudfounding for this specific feature could be opened ?

1 Like
#12

Such a feature is known in different synchronization applications under quite different terms: Selective Sync / On demand sync / Smart Sync / Placeholder file sync.

The discussion around this feature came up often over the last years and maybe would benefit from bundling this information together a bit. The use cases can vary a bit in the details, but the general structure is almost identical.

Here I give one case from real-life example:

Given is a directory structure such as

/projects/project_a, [7GB] /projects/project_b [11GB] /projects/project_c [45GB] /projects/project_d [3GB] […]

With each containing a sub-hierarchy such as /projects/project_a/data/raw/component1/image_0002.jpg to …/component7/image_0768.jpg /projects/project_a/data/processed /projects/project_a/summary/metadata.xlsx

This directory structure is shared among multiple colleagues both on-premise and on the road. At some point this grows beyond the available disk space. Here currently syncthing allows with the ignore files to exclude certain folders to reclaim the disk space, other sync applications allow using such a “selective sync” feature to free the disk space as well. In the meantime my colleagues might change the directory structure slightly, e.g. moving images from one component folder to another or similar.

Now I receive a request from a colleague to “Please check and update the metadata of project_c by looking into the last images”.

Here using “on-demand sync” I go into the folder, open the metadata file and start my way up from the last image opening them as necessary, this requires only a few hundred MB of disk space and transfer quota.

As far as I can see this is currently in syncthing very difficult to achieve using manual manipulation of ignore files without downloading essentially complete projects to access (read and write) a few files. I don’t know the specific file or folder names but I can navigate there quickly. It also improves collaboration as I can see if someone added files/moved/deleted files without having to sync the full contents.

Such a use case clearly requires a sync application and is not covered by a web server as it is essential to have a two-way sync between the colleagues on the road and the office. On the road both disk space and available bandwidth can be much more restricted so other syncthing features such as a differential sync are very handy.

To implement such a feature a relatively deep integration into the specific platform is required. A few years ago this would have been very challenging. Nowadays, under names such as “cloud files API”, “Finder Sync extension” there is platform support (at least on Windows, Cloud Sync Engines - Win32 apps | Microsoft Docs, and MacOSX, macOS File Provider API for Selective Sync? - #3 by calmh). Under Linux the FUSE-approach is a potential approach as neither GNOME, KDE or other desktop environments contain a similar API.

Given that there seems to be quite a lot of interest (approx. 50 issues on the bug tracker, >50 forum threads) and even a teaser by Jakob from a few years ago (Dropbox Infinite - #9 by calmh) it would be fantastic to get an update on the thoughts from a developer perspective, especially with a focus on what the humble users can do to remove potential roadblocks.

1 Like
#13

Its absurdly involved feature, with caveats, bells, whistles and more on each platform.

The current blocker is that nobody wants to work on this.

#14

I came up with a simple idea which would be a good starting point:

there should be a default for folders where (aside for the machine you used to put the files in the folder) the .stignore file lists (and ignores) all the files contained.

Hence when you want to “download” just a file, you just have to un-ignore it from the list that you already find in the desired folder. And when you are finished with a file and you want to delete it from the local machine you “ignore” it again in the list and manually delete it.

Sorry for the bad english but it isn’t my first language. Ask questions if I wasn’t clear.

#15

Yes it would be fantastic to hear from developers about this feature that (imho) would boost the usage of the app a lot.

#16

Is that not just a * at the end of the ignore file then?

#17

but you need the list to be automatically generated and updated everytime you change something in the folder

#18

A tool which work with the Ignore List to implement a selective sync is

https://github.com/galilley/syncthing-pyselective

but I tested it only, I have no deeply experience with that

#19

I’ve come to this thread because I am also looking to sync selective files, especially on mobile devices.

I would like to have a documents / music files on a NAS and be able to sync (make available offline) some files when I go on vacation / travel.

I believe the feature could be implemented similar with Ignore Files but as “Synced files”.

As I see it, Syncthing currently syncs by default since this is it’s main purpose. An option could be to mark a folder as “do not sync by default and only sync the file patterns in this list”.

The list can have same format / semantics as “Ignore list”. When the list of “files to sync” changes, local files are not removed. The current feature to “Revert local changes” could be used to remove local files that should no longer be synced.

WDYT @calmh ?

#20

That’s what ignore patterns already do:

!/filetosync-1
!/filetosync-2
!/nested/filetosync-3
// ignore everything else, i.e. don't sync by default:
*

Edit: Added the pretty crucial ! I forgot in the first iteration - thanks @acolomb for pointing it out!