Immutable repository

Feature
1

Hello everyone! I’d like to discuss a feature, I find very useful.

There are types of files, once created they never change. For example, I have photo collection, organized like this:

… 2012 -01 –file1 –file2 … –fileN -02 … 2013 …

In such a structure once added new photos I will never delete or change original copies. With help of syncthing it is cool to use several devices to add to collection. But I want to be sure every photo is safe, and not accidentally deleted. Ofcourse, file versioning is very helpful here, but every time it requires additional attention to ensure everything is ok.

This is just an example, there are other cases: archives, git bundles.

So, what I suggest is to make repository immutable, unlike “Repository master” every device (user) can add files, but no one can delete. As additional feature it is useful to be able to check integrity of this kind of repositories.

2 Likes
2

So every file gets to be synced once, and only once, you mean?

3

I mean initial state should be maintained, if one node deletes or modifies files (maybe unintentionally), original files should be restored. Same in case of integrity check failure. By the way about integrity check, I mean user can initiate it.

4

Right, initial state. But then you want to accept new files, and immediately lock those files down for future modifications. Like a write-once medium.

5

Yes, good point, exactly like write once medium (multi user and very reliable).

6

I didn’t see the usecase for a “write once repro”…

e.g. DSLR photos: what happen if i add tags to one photo? It will be ignored everywhere.

I think we net more a “counterpart” to “Repository Master”: maybe something like: “ignore changes”…

e.g.:

Node 1 - repro A Node 2 - repro A with “ignore changes” ON Node 3 - repro A

So i can do changes on Node 1 and 3, that will sync to all nodes. But if Node 2 change something, that will be ignored on every other nodes. e.g.: Node 1 or 3 change file A, this file will be sync on all nodes… Node 2 change file A -> ignored. After this Node 1 or 3 change file A again, then the file will be overwritten on Node 2, too.

EDIT: Maybe a other solution: On every Node you can set beside “Shared with Node XY” a “read-only” flag. Because with this you can “protect” the Nodes 1 and 3 from changes of Node 2, e.g.:

Node 1 - repro A shared with: Node 2 with read-only ON and Node 3 with read-only OFF Node 2 - repro A shared with: Node 1 and 3 with both read-only OFF Node 3 - repro A shared with: Node 2 with read-only ON and Node 1 with read-only OFF

7

As I understand what you are describing in “EDIT” is more general case. If every node “ignores changes” from every node we will get what I’ve talked about. I guess implementation will be quite complicated, as diferent nodes will have diferent “up to date” files, evaluate this is up to Jacob.

8

And from the begining I thought of this as option. This way if one day you suddenly need to change two years old photos tags you can turn this off, set tags, syncronize all nodes, turn back on. This procedure should be rare.

9

The “sync each file once only and don’t accept any changes” feels a little bit too niche for me right now. I’ve imagined the following “modes” for a node;

  • Full sharing: the current default, updates flow in and out.
  • Repository Master, currently implemented: files that are updated locally are published to the rest of the cluster. No changes from other cluster nodes are allowed.
  • “Untrusted” or something, where a node receives changes from the cluster but is not allowed to contribute to the cluster state. Not implemented.

For the later, some awesome encryption would be nice, there are issues for this on github…

10

I have another idea for this:

similar to repository master, but if another node sends a change you get a message on the webinterface like “node xxx has changed y files, apply changes [y/n]?” also with a button to show a list of the changed files. maybe even sync that this change is applied to other nodes that you only have to accept the changes once

11

@calmh, untrusted nodes is cool feature if you want to create kind of “cloud” nodes. None of the listed modes solve the problem I am talking about. Don’t treat me wrong, I appreciate your intension to keep things clean. But I’ve just tired of unintensional modifications done by dropbox, btsync, etc. Personally I met the situation of sensitive data corruption, moreover, versioning system didn’t help as lot of time passed (I am talking about you, btsync).

What can I add here…, lets look from other side. Pretend you didn’t heard of ZFS, BtrFS, Tahoelafs, you want to save your files reliably, using your own devices. What options do you have? Silence…ah no you have one, you can create a torrent and seed it to several devices, then anytime you can hit “check hash” to ensure your data is safe. Ok this is quite rediculous :slight_smile:

The truth is there is no handy tool for normal people. Syncthing could give an answer.

As for @Alex idea, seems good to me that kind of master device can confirm or reject changes, made by others or itself.

12

Seems to be nice… But this is IMHO only manageable if only a few changes exists. Change hundreds of files results in a more and more confusing situation… IMHO it’s hard to create a web interface for it, that is good for a few changes and at the same time good for many changes in short periods.

btw. we have this related issues yet:

13

Oh, yeah, I realized that and didn’t mean to imply that they did. It was more a “this is what I’m thinking” kind of post. There are probably many more “modes” that might be useful; the ones I listed, your suggestion, the “ask for changes” mode suggested above, various kinds of “append data to files, but don’t modify existing data in the files”, “allow all changes but not deletes”, etc. Maybe it should be a long list of checkboxes of allowed operations. Some of them are stuff that ought to be enforced by the cluster at large (i.e. when a node is not supposed to be allowed to originate deletes), which is more complex.

In the end, I’m getting into hard prioritization (have you seen how many valid and useful feature suggestions there are?), so for me this is probably not going to happen in the close future.

14

Ok, priorities are important. Maybe some day this will be part of more general concept.

15

This ‘write-once’ repo would be ideal for a situation I’ve been struggling with for a while (using DropBox and BTSync). We have a big project and several people donate files to the project. Once donated, the file is permanent part of the project and noone (except for the repository manager) should be able to make changes to this file, or remove it, and so on. Until now this requires the kind cooperation of all people involved, but people make mistakes and too often files get lost and must be recovered from archives, which is tedious. A write-once repo would solve this perfectly.

16

Yes, in general there is no easy way to make sure that what you get out of repository is what you have put into. And that is true for every sync software.

17

I do see the use of the first posters question, I am looking for the same:

i.e. sync all images to ONE master repo THEN only do 1-way sync, i.e. newly added pictures on any other node will sync to this master but no deletes should sync through. This way I have a safe copy of all files on the master repo no matter if I delete images intentionally/unintentionally on any other client as they will not get deleted from the master repo. Is this doable somehow? (kinda looking to archive certain folders for long term storage)