Distributed versioning/backup with Restic and Syncthing

(Matthew Palermo) #1

Restic seems to be a versioning / backup / deduplication archive system that could work harmoniously with Syncthing. Restic’s purpose is simple: Store many versions of a directory (or multiple directories) in it’s deduplicating repository format. It has been designed specifically to allow for parallel writes. This is possible because storing a new backup only creates new files and files are content addressed so that if two files have the same file name, they must contain the same contents, thereby avoiding possible merge conflicts. It seems then, that Restic should work well with Syncthing, even with multiple devices writing to the repo while disconnected from each other. Contrast this with git which can cause headaches or data lose if Syncthing merge conflicts happen.

I haven’t tested this yet but I am so excited about this find that I wanted to share it preemptively. I have discussed this with the Restic developer and he also thinks this should work well: https://forum.restic.net/t/restic-parallel-writes-with-syncthing-or-dropbox/797.

(Matthew Palermo) #2

I found that chunk based compression which is the basis of restic doesn’t perform well for many small changes such as changes to a text file. Delta encoding such as that employed by versioning systems or solid compression is much more space efficient in this domain. I’m now dreaming of a parallel writable repository which uses delta encoding and/or solid compression.

(Bob Hope) #3

@jerryjacobs mentioned borgbackup here:

Might that be an option for you?

(xor-gate) #4

Borgbackup has the same ideas as restic, but restic is less old and widely used. It is written in python and C instead of Go. If you can make a choice you could better probably go with restic if you ask me. But borgbackup is running for a few years now on the servers I manage and works fine, because at the time of deployment restic was less mature.

(Kluppy) #5

Just FYI. My (very limited) testing of restic found it a little resource heavy for a small VM. I don’t think borg failed on the same size VM.

(xor-gate) #6

It is possible restic is less resource friendly than borgbackup. But real testing and creating a report and give feedback to the project has not been done (as far as I know).

(Matthew Palermo) #7

I think BorgBackup relies on file locks to facilitate parallel writes to it’s repository and so it doesn’t allow for Syncthing compatible parallel writes like Restic. One day I’d like to try hack a program which allows Syncthing compatible parallel writes like Restic but uses delta or solid compression in some clever way.