Is it possible to make Syncthing implement based on IPFS?

#1

Hi Syncthing Developers,

I am a user seeking a reliable and flexible solution for private storage. Recently I learned a few things about IPFS (which is thought as a replacer of HTTP). I find IPFS shares a lot of common underline ideas with syncthing. Currently syncthing is great for data files backup and sync, and has windows, mac, linux and android platforms support. But I think if syncthing is built on more general infrastructure, it will have more opportunity to move to future.

Any idea? I would like to hear your thinkings.

Thanks,

(Jakob Borg) #2

I think the two concepts are rather different, and each serve their niche.

#3

Can you explain deeper?

My understanding is the technologies are same, and infrastructure are same, but the applications built on those technologies and infrastructure are different.

(Jakob Borg) #4

One is a distributed filesystem, the other is a file sync application. No technology is shared, afaik.

#5

Hi Borg,

Here is a project brig for your reference.

brig - decentralized & secure file synchronization

https://brig.readthedocs.io/en/latest/

(Audrius Butkevicius) #6

Sorry, why is this relevant?

#7

brig is also a file synchronization tool, but based on ipfs, which naturally with distributed & secure & version control functions.

(Audrius Butkevicius) #8

Right, so what are you after on this forum?

#9

Hi Butkevicius,

I am a user of Syncthing, and currently I think Syncthing is great and your guys do a great work. But when new technologies comes out, and it looks having a bright future, why not embrace it.

I am software developer too. I believe as a developer we all want what we do are meaningful. Making self be well-informed while working hard are always right.

I just want to hear some opinions about IPFS for file storage (include sync, backup applications) from the first-line developers. Thatā€™s why I raise this question. Why not open-mind to take a look and think about it.

(Herbert Snorrason) #10

A possible reason: Because appearances arenā€™t everything. Iā€™ve seen multiple content-addressed distributed filesystems that planned to replace the web; Iā€™ll certainly consider the possibility that this time around it might actually work, but Iā€™m not holding my breath.

This is the key thing here. IPFS is for file storage; Syncthing is for file synchronisation. If you can usefully have a mountpoint for IPFS, then Syncthing should be able to use it for a directory. If you canā€™t, well, thatā€™s a bit more complicated. That said, content-addressable file systems are extremely different from our usual ways of thinking, and I suspect that most of brig is actually dedicated to storing the metadata needed to build a file tree.

The one way I can think of that would make sense for Syncthing to do with IPFS would be if you could use Syncthing to synchronise things that arenā€™t files on your local filesystem - if there was a system through which you could have a node that stored its files on S3 (or, if youā€™re feeling particularly ironic, Dropbox). Then one of the ā€˜backendsā€™ could use IPFS.

Mind you, by my understanding that would mean publishing the data so that it is accessible to anyone who gets hold of the IPFS address of the file. So probably not the best thing for private data, unless you store it all encrypted.

(Audrius Butkevicius) #11

There are many great things out there.

People using syncthing care about syncing files between their machines, I donā€™t see how IPFS fits here, other than wanting to wedge in a kitchen sink into syncthing.

We might as well sprinkle a bit of blockchain too because because itā€™s cool new and all of that and weā€™re supposed to ā€œembrace itā€.

Syncthing has a filesystem abstraction, so if you care strongly, and have the skills (given you are a developer), knock yourself out and add a blockchain based singing dancing filesystem.

I personally have no interest in this.

(SiSa) #12

I like the idea and wonder how mankind would have evolved, if we all had singing dancing filesystems since the 1990sā€¦

(Jakob Borg) #13

IPFS is a fine distributed content addressable file system for when you need a distributed content addressable file system. Restic is a neat backup system for when you need a backup system. Rsync is a great stateless one way sync program for when you need to sync files in one direction without keeping state. Syncthing are none of these things and thatā€™s fine, because they exist on their own, and none of them are Syncthing.

#14

As a distributed system, synchronisation is a must have function. So naturally IPFS supports sync. There are two common things I saw between syncthing and IPFS.

  1. Network: they all use p2p transport between nodes.
  2. Block Exchange: they all exchange files based on block.

The things that syncthing can leverage from IPFS:

  1. IPFS Routing layer includes
    • peer routing ā€“ to find other nodes
    • content routing ā€“ to find data published to ipfs
  2. The merkledag data structure, which I think is a little more advanced than index and index_update in Syncthing.
  3. Naming ā€“ PKI namespace and mutable pointers(https://github.com/ipfs/specs/blob/master/architecture)

You can find more detail in https://github.com/ipfs/specs/tree/master/architecture.

You can setup private distributed storage network on IPFS (https://medium.com/coinmonks/setting-up-your-first-distributed-private-storage-network-on-ipfs-part-1-a6ff15222b90)

(Jakob Borg) #15

If you want to try to re-engineer syncthing on top of IPFS, go for it. We, however, will not.

(Herbert Snorrason) #16

Thatā€™s not the same thing as being a synchronisation system. IPFS distributed content-addressed files across the swarm. You still need a system to keep track of the mapping between file names and contents, and synchronise that between nodes.

Advanced is a red herring. The F-35 is a lot more advanced than other jet fighters. It remains entirely an open question whether it is fit for purpose.

Alright, but at that point, what advantage does it bring over Syncthing? It appears to me that such a use case is seen as decidedly a secondary objective, in light of the ā€œreplace HTTP with a distributed protocolā€ focus of the projectā€™s website.