Lowlevel API for structured data

(Benno Fünfstück) #1

Would it be possible to extend syncthing to support more than just files? I am thinking about ways to do so, for example adding a lowlevel API / plugin interface to directly sync some blocks (not tied to files).

For an example application, consider building an Etherpad-like synced pad that uses syncthing to distribute the database among the peers. Just using syncthing to directly sync the db is of course going to result in conflicts, but if there was direct access to the block-level interface of syncthing then the pad could build it’s own conflict resolution using CRDTs on top of that.

Another application could be a syncing other structured data where you want to use CRDTs or OT to avoid conflicts such as passwords or configs.

1 Like

(Jakob Borg) #2

It’s hard to say because I’m not really sure what you want, exactly, but my guess is no.

Pretend you had such an API. What would you ask it to do?

0 Likes

(Benno Fünfstück) #3

Sorry for the vague question, I realize I don’t know exactly what it would look like myself :slight_smile:

I think the basic primitive that is required would be a kind of append-only file with realtime sync (so like a channel). That file would never have conflicts, since merging can simply be done by adding new items from both sides (the channel would not guarantee ordering, that is something that an application on top has to deal with itself).

0 Likes

(Jakob Borg) #4

You can have that API today - it’s a directory with files, each file named by its creation timestamp. (Append some unique ID if you’re worried about timestamp collisions.)

1 Like

(Jonas L.) #5

I could think of (in the far future) Syncthing as an way to setup protected connections (like SSH but using the discovery server and relay servers). So that the “server” can share some services (ports) which the “clients” can make available at them (comparable to SSH port forwarding).

Very nice would be git over Syncthing, but this requires atomically updates (for updating the refs to the last commit and for “git gc”), but by such remote socket connections, it would be at least possible to easily sync projects between machines.

0 Likes

(Matthew Palermo) #6

@bennofs and @l-jonas : I’m kinda curious about this as well - that is using such a user friendly tool like Syncthing (or even Dropbox like tools) to help us Sync database-like data - because it’s just so damn user friendly (for networking).

@bennofs, have you considered using the file system like @calmh suggested? Check out the backup program Restic and my post about it here: Distributed versioning/backup with Restic and Syncthing. It is the only program that I’ve found so far to use a Syncthing friendly, file-system based, database format. The advantage of doing this is that the program will work for any file syncronisation system like Syncthing, Nextcloud and dropbox. I’m thinking about mucking around to create something similar using a different compression method one day (suitable for text files) and much more explicitly Syncthing friendly.

@l-jonas, I agree, it would be awesome to be able to tunnel TCP through something like Syncthing: Using Syncthing network resolution for other tcp programs?. I looked at many different P2P VPN solutions but I have never found a simple solution that would work for me and all my different varying operating systems. I looked into Tor, I2P, tinc, Hamachi, ZeroTier, FreeLAN, n2n, cjdns, VpnCloud, PeerVPN and P2PVPN.

0 Likes

(Jonas L.) #7

The model of http://couchdb.apache.org/ would work with the filesystem based approach (filename: [document id]_[revision id] (the revision id starts with a version number)) but a normal SQL database (which provides consistency) would not be possible (consistency and availability aren’t possible at the same time).

1 Like