Integration with Storj for Decentralized Storage

Feature
1

I stumbled onto Syncthing because it was mentioned in a recent article.

Storj is a crowdsourced P2P network that allows encrypted and private storage. Very similar goals to Syncthing. We figure if the data is encrypted client side and split into pieces you can store it on devices other than your own. Since the data is stored hundreds of independent devices, and not a centralized service or server the user retains control over the data. If you would like more technical details feel free to read our whitepaper (bottom of our website).

Wanted to figure out if there was interest to integrate Storj into Syncthing. If there is interest from the community we would be happy to provide support, bounties, and/or sponsorship to help get it done. Happy to answer you might have.

5 Likes
2

Looks like I can’t post more than one link as a new user:

3

Synching does not support sharing data with arbitrary devices. Each relationship needs to be explicitly approved from both sides, which I think does not fit in with your model.

2 Likes
4

I would define the relationship to be between the user and the network. We offer an abstracting service that treats that as one, so you don’t have to individually deal with each relationship.

In my mind Storj would appear as a single device that you could sync with. For people with limited amount of devices or bandwidth/storage I think it could be a very useful option. Only the user would have control of this data because neither the service or network have the keys, proper mapping, or metadata associated with the data.

The integration could offer singular relationships between devices, but I think that would defeat the purpose of decentralization.

5

I found about storj a couple of weeks ago and I’ve been playing with it¹. I like the idea of shared encrypted storage on the internet very much, specially after watching the Node Summit talk: The cloud is Dead - Long Live the Fog!. The cloud sucks, but these people are hacking it in a cool way.

Syncthing is better for me than other ugly closed things like dropbox because I have full control over it and I decided that my data must never leave my home network. However, it sounds appealing to keep a backup in the fog, as they call it, for a few of my things. They will never be readable by anyone because the decryption keys will not leave my home network, and it gives me guarantees that I can’t have with my small set of relatively co-located devices.

The goal of both projects sounds complementary. And an integration like the one proposed certainly would require work and will possibly encounter unforeseen difficulties, but it makes sense to me.

There are other projects like IPFS, so this work could open the door for some crazy new ideas.

¹ Of course, there’s a snap: https://github.com/Storj/storjshare-daemon/blob/master/snap/snapcraft.yaml :D:D:D

3 Likes
6

FYI there’s also Sia, which is really similar to Storj (from my understanding). However, Sia already has a working version released.

7

Sia is pretty cool, but Storj is further along. It stores about 10x more data, and recently came out of beta. Plus I don’t think most people here want to download and sync a blockchain, or buy cryptocurrency which is required in Sia but not Storj.

1 Like
8

There is also MaidSafe which is direct competitor to Sia and Storej and then there are things like infinit.sh which alows to create decentralized storage network on your infrasturcture with your defined redundancy, dedup, encryption… it’s only problem is that it is not opensource : (

1 Like
9

@super3 I wouldn’t mind syncthing compatibility with storj, more options the merrier.

2 Likes
10

Did this happen? I’m looking for an open source Dropbox replacement, and the combo of Syncthing and Storj seems like it could be that… (Correct me if I’m wrong!)

11

It did not, and will not.

1 Like
12

Hi there!

I have been using Syncthing for years on all sorts of devices and I really love it. Thanks to all the people responsible for creating and maintaining it!

I have recently started using Storj and thought about the possibility of having it as an untrusted (encrypted) device, a feature that was not available on Syncthing back when this thread was first created. The idea was to have Storj as some sort of backup for folders being shared through Syncthing.

But I see from this thread that the idea of a Syncthing integration with Storj was not well received by the Syncthing community. Would it be possible to elaborate on why this is not a good idea?

Thank you for your attention!

13

I don’t think it’s decided that it’s a bad idea; from my side it just look like an ill defined idea, and no one has put in the legwork to define what it means or how it would work, or why.

3 Likes
14

Why

The ability to store files in the cloud greatly increases the reliability and availability of data.

Imagine that you are going abroad. You take a synchronized laptop and phone with you, but you are not sure that the Wi-Fi in the hotel will work, that the mobile network will work, and even that there will be sufficient access to electricity. You cannot predict how often you will be able to connect your laptop and phone to the network at the same time.

At the moment, the best solution would be to use Dropbox instead of Syncthing — Dropbox will be predictably online when you connect at least one gadget. But it would be great if Syncthing could be relied on in these cases as well.

How

This can be implemented if, in addition to Devices and Folders, we add a third entity, say Folder-in-Cloud.

Folder-in-Cloud can be set up by providing credentials required by a cloud storage (Storj, Dropbox, so on), and pointing the path in the remote storage. For example, we could configure Folder-in-Cloud named “MyDocs in Storj”, which refers to particular path in particular Storj bucket.

Folder-in-Cloud is not a Device, because it is passive by its nature. It can be read or written from a local Syncthing instance via cloud API. But the cloud does not run any specific Syncthing software, and even not aware that it is used by Syncthing.

Folder-in-Cloud is not a local Folder, because we never have direct access to it. It is always something remotely accessible, but never here.

To use Folder-in-Cloud we turn on synchronization between our Folder “MyDocs” and Folder-in-Cloud “MyDocs in Storj”. So our real filesystem folder on particular host gets two-way sync with the cloud. It is done by local Syncthing instance. This does not directly affect other devices.

The Folder-in-Cloud may be displayed next to Devices list in particular Folder settings. We can sync Folder to other Devices and/or to other Folders-in-Clouds.

Probably each Folder-in-Cloud should be compatible only with on id of ordinary Folder. So we can sync “MyDocs in Storj” with “MyDocs”, but not with “MyPictures”.

It would also be great, if after creating and configuring “MyDocs in Storj” on one Device, we could share this Folder-in-Cloud (i.e. configuration settings) with other Devices.

15

I think that’s Object Store (S3) backend · Issue #8113 · syncthing/syncthing · GitHub

1 Like
16

DQB029JVwAALyEu
DQB029JVwAALyEu1920Ă—1415 78 KB

Also, you can just run a t2 micro instance (or whatever cheapest tier instance in other providers), run syncthing, and have your “folder in the cloud”, no need to get on the blockchain bandwagon and pretend that it’s somehow superior.

2 Likes
17

I believe that Storj or Dropbox are still superior. Because from the moment of sign-up they require literally no additional actions from the user, and still provide almost 100% uptime.

Any do-it-yourself virtual server depends on your personal readiness to repair the server at unpredictable moments.

Also, free versions of clouds superior to any “cheapest” solutions, because their near-100%-uptime does not depend on the reliability of payment systems and your financial discipline.

18

Why not just use dropbox client to talk to dropbox then? Why does syncthing need to come into the equation?

19

For me, any cloud services are rather niche solutions. I would store the data of some folders there (as untrusted nodes), but leave Syncthing to manage all the synchronization.

No cloud can replace the advantages of Syncthing — openness, cross-platform, client-side encryption, no restrictions on data size.

Even while traveling, temporarily switching to pure Dropbox is very inconvenient because we lose client-side encryption and have to rethink our willingness to store every file in the cloud.

Other clouds (except Dropbox) are usually not even an option. Professional clouds (Storj, Amazon) don’t try to be usable on Android phones. Consumer-oriented clouds (Google Drive) often do not have user-friendly and functional Linux clients.

If Syncthing worked cross-platform with some cloud APIs, it would solve a huge problem.

1 Like
20

Then just do it? If you don’t want to have the files visible (clear text) in any particular “cloud” then you don’t want to use any of the search, edit, share, picture albums, etc. features from there then it doesn’t make sense to insist in using Dropbox or Google Drive or Google Photos or any other particular service. You could just as well use any service where you can simply run syncthing, like a VPS (they start from literally free, like Oracle Cloud with up to 200GBs). Or have a look at https://lowendtalk.com/ there are plenty of choices to pick from.

Other than that support for some of the mentioned backends is horrible, as far as I know the API for Google Drive doesn’t allow for changing the files (either in the middle or appending to them). If you have some big files and flakey connection it’ll move syncthing from the thing to use for this use case to just something that eats all your network and battery forever but it doesn’t finish because it starts all over again all the time without getting anywhere.

1 Like