Selective download and placeholder files

This is a great idea, but this is WAY far ahead of where the project is at the moment. This is not worth your effort yet… Why? Well, the #1 reason is such a feature has GOT to be cross platform… And that means built into the protocol – so it works in Java, GTK, Web, whatever.

(The ONLY way I see to do it now is if you piggyback on the BEPv1 protocol using the read-only file permissions flag). And that’s not a good way, heh.

Right now the protocol doesn’t support your idea, although I like your idea. I am in favor of writing a BEPv2 protocol … I’d even do the work for it… Anyway all that aside…

To do that in GTK, it means we’d need to put it in a hacky revision of the BEPv1 protocol, which will contain the data necessary for on-demand drive mounting (ie. like Bitcasa). and sharing of preferences among clients.

i’m on board, but not until the basics are worked out…Namely, protocol redesign. But I think #1 priority is take stock and fix existing critical bugs and UI issues. Then redesign later.

There is no need to change anything in the protocol, so not sure what are you talking about.

As @AudriusButkevicius pointed, there’s no need to change the protocol. This isn’t a new feature, but a different way to interact with the current available selective sync and to solve one of its main issues: currently you can’t to know the content of the folder to choose what to download before download it.

It isn’t the only way, but it was proved a good one, by SkyDrive implementation, and we have the chance to avoid Microsoft mistakes.

First off, thanks for syncthing. It looks like it’s going to be a very useful program for me.

Re this thread: I think something along the lines already discussed would add value to syncthing. I see two, among other reasons, one would use syncthing. The first is as the name suggests, replicating a store elsewhere with the primary benefits of a backup in which case full syncing a’la rsync is necessary (though I’d prefer to use rsync for this purpose).

The other is sharing a store with others or just yourself over a number of devices. In this case full replication is not necessary, and when it comes to sharing to devices such as mobile phones it is not even possible. In this case, .stignore is useful, but clumsy as one still has to traverse the whole directory tree to see what has or hasn’t been copied over. For this use case I wonder if it would be possible to implement some kind of caching system, where the files that are synced show up in this cache for easy listing, use or deletion when media space runs out.

First of all, i must say i dont know internal work of syncthing and i am clueless in GO but i feel like my proposal can be done by simply GUI tweaking. Basicly, first step before true “selective sync”, can be done by just modifying GUI.

When user is creating new share:

  1. add “ignore all” directive to all newly created folders by default (in last few days i created many shares, and in many cases i didint want sync to start right away. I wanted to set up my ignore list etc.)
  2. add “sync ON / Sync OFF” toggle button (on screen where you create share and on list of shares) that would effectivly remove or add ignore directive for a share
  3. merge list of ignored files with list of files that are not ignored, so user has directory like tree in gui on which he can press “sync ON” or “sync OFF” to every file / folder which only effectivly removes that file / folder from ignore list.

Would that even work?

Just a heads up to @kozec that /rest/tree now exists, given he has any interest in this ;). Selective sync to follow shortly.


[quote=“AudriusButkevicius, post:43, topic:1779”] Selective sync to follow shortly. [/quote]Does that means you are going to implement it as feature in Syncthing, so it wouldn’t be required to do hacky things with .stingore?

Yes. I have basic prototype for the UI, need to actually plumb it into the scanner etc. I am sure feature requests will follow asking to sync selections etc, but that’s a different story.


This sounds excellent. I look forward to it. :slight_smile:

To clarify what I meant in my most recent post. And this is one idea which attracted me to come start developing on this project, to learn if this is possible, and how long it might take… It’s the same sort of main idea as ‘selective sync’ that people like in Dropbox…

My inspiration is instead from another Bittorrent like file sharing protocol called Bitcasa. (Closed source and costs money , company is being sued, etc which is why I dont use it)

What I was talking about is to have Syncthing have ‘streaming’ behavior like Bitcasa. As an option. Suspend disbelief for a second, and imagine this is what I’d like to happen… ie.

(1) i have syncthing runnng on my home computers and a VPS or EC2. And they are all sync’d up.

(2) I go to my girlfriend’s computer. I install Syncthing We want to watch some movies / tv shows that are on syncthing. But I don’t want to (A) download all the shared files, nor do I (B) want to download the movie prior to watching.

(3) So essentially, the behavior in my pretend example works like this… Syncthing creates ‘placeholder’ files for the entire filesystem tree on my girlfriend’s computer, but doens’t populate them with any data. I see them as drive E: in windows or whatever.

(4) I naviate to E:\Movies\ and click on IronMan.mp4 or whatever. At this point, VLC issues a blocking read to E:\Movies\IronMap.mp4 . Syncthing either detects or intercepts this (just like Bitcasa does) ,and starts downloading the file. After a certain amount of pre-defined read-ahead occurrs (essentially a video buffer), then VLC’s blocking call returns, and IronMan.mp4 starts playing.

(5) Thus, two benefits overall (A) I can see an entire massive shared filesystem and browse it without downloading anything other than Filesystem metadata (how thing’s are layed out , what the files and folders are, etc). … and (B) There is some sort of mechanism or driver that allows me to ‘stream’ files.

(6) In the ideal implementation , #5 doesn’t just start ‘streaming’ the file from Block 0. It starts streaming the file from the point of fseek(File *fp, 0L, SEEK_CUR) … Or in other words, the current positon . So if I click on IronMan.mp4, let it buffer and load, then I skip ahead to the halfway mark… Then Syncthing will detect a read halfway through and start populating the file from there, or start writing the file at the zero mark from there, etc.

Okay, so I’m not familiar enough with video formats to know whether this is possible with .avi and . mp4 formats, or whether this is a pipe dream that only Netflix and Amazon Video can achieve. But the basic idea is to emulate the behavior of Bitcasa…

Meaning I want the E:\ drive to act as a ‘download-on-demand’ drive at the very least. Where Syncthing only downloads a file when I try to use it (meaning open(), read(), etc. And even better, Syncthing somehow only downloads the relavent portions of the file.

Just the same way that Synchting already only shares the parts of the file that have changes on write(), I’d like a similar efficient process for read() – at least as an option .

That’s my dream for ‘really cool feature that’s difficult to implement’ – it’s essentially like turning Syncthing into the mailman for a Network Share… Like how NFS or Samba behaves when mounted as a Drive letter… It only downloads on demand.


Wow! This is interesting, @cydron, but I’m afraid is too close to what Microsoft did on SkyDrive and fail: the users don’t have a simple way to know if a file is “real” or just a placeholder.

This is the reason I suggested an easier to understand and, thanks to the explanations of the developers, easier to implement. BTSync implemented it well, for what I can see.

May this could be a evolution of the placeholder files? So, we could watch how users react to the feature.

Oh really? Man I better go read up on Skydrive. Good call. That idea is on hold until I figure out why it failed for microsoft. Wouldn’t want to waste time on a feature that has no demand.

Sure, maybe copying BitSync’s style is better. They are the competiton , as I see it anyway. Thx for the feedback.

IMO, just making syncthing not require full syncing, preferably in a better way than btsync (currently you have to wade through the folder hierarchy of the source being synced) would make syncthing more useful for sharing with devices that have storage issues like mobile. Having the ability to view your synced files in a cache rather than the source hierarchy would be a killer option.

Not even sure what you mean by that. Not familliar with how btsync does it.


Sorry for being unclear in my description. Here’s what I was trying to say:

syncthing and btsync work almost the same up to the point where the shared folders have been defined.

From there on, syncthing starts syncing the shared folders - it does this by starting to copy all files. Of course, on a target device with less free space than the source size, you eventually run out of space.

btsync, on the other hand, only syncs the folder structure of the source folders, leaving it to you to pick the files or folders that you actually want on the target device, thus avoiding the syncthing problem. Files that are downloaded have check marks against them to indicate local availability. This is nice, but the problem is that in order to see files that you may have synced, you have to browse through the folder structure, folder by folder - which is a pain.

So while we have to maintain the same directory structure on the target device in order to allow full syncing, it would be nice if syncthing could start by only syncing the folder structure (like btsync) and then add the feature where the files that are already synced are listed separately for easy access.

Not sure if I’m any clearer, but hope that helps.

I think that OneDrive has solved this problem properly. You simply choose which folders to synchronize, end of story.

I think that what @kinleyd wants is to be able to see a list of already synced files on Syncthing interface and to open them from there. Am I right, @kinleyd?

1 Like

Exactly, @robsonsobral.

Actually, if we were to be really careful and select only folders that fit the space available on the target device, then syncthing has also already solved the problem. I guess the functionality I’m looking for is simply having access to, say my whole collection of music and videos, but selectively downloading what I want without worrying too much about space availability nor having to go back and create new shared folders on the client and the master in syncthing everytime I want something.

1 Like

Well selective sync solves this doesn’t it? You select folders you want. Once you are done, you unselect them (and perhaps part of that we delete the unselected files) and select something else?

How is seeing files which are synced in Syncthing any more beneficial than using the explorer?