Partial Syncronization

Surely at some point this feature will need to be made, so I might as well bring it up and maybe send a PR later.

As far as I can tell, when a node has a repo it gets the entire repo. I’d like to replace DropBox with SyncThing + my Linode and while that will work it sadly will not work for my phone, as I currently have about 8G of data in DropBox. What I’d like is either

  1. On demand retrieval of files (like DropBox)
  2. or Selecting files/folders to syncronize on a given node (like BTSync)

I suspect the latter is more feasible since it fits the ST model more, but the former is nice since you don’t have to have ST running all the time on a device that has to deal with battery, ram, cpu limitations.

Any comments welcome. I’ve been writing go lately a bit and will try my hand at helping out, though I doubt I can jump in and add this feature easily as it’s probably pretty significant.

yep :wink:

That’s awesome! :slight_smile: So, when two nodes connect they exchange indexes, for all files in the repo. But a node doesn’t have to sync all of them. I’d probably start with something that acts as a filter on incoming index announcements - if a file doesn’t meet the criteria, just discard it. The files in the index that are accepted will be synced as usual.

A downside, if only an aesthetic one, is that the node will be shown as permanently out of sync by it’s peers since it will only announce 56% of the repo or something like that. This is the truth but could probably be shown nicer. The protocol has the concept of “mode flags” per node & repo, so a node could announce that it is only a partial syncer and other nodes could understand the situation.

Then there’s building a GUI tree view file picker thing.

But this is not the best solution, isn’t it? Because the hole index must be send to all nodes.

Better solution is IMHO:

  • On Node A the user sets the directory he wants to sync.
  • Node A sends the directory list to Node B
  • Node B sends a index with the filtered files information back

But here a database instead of the flat index file will be helpful, see:

This on the other hand “outsources” the filtering work to all other nodes in the cluster and requires protocol changes.