Adventures in storage backends

Here’s an update on what’s going on with the “persistent index database” feature. I’ve been trying to integrate with the various embedded data stores I’ve found, and most don’t work for our purposes. Here’s a list/log of the packages I’ve tried and how it goes.

Wont Work


Full featured, stable, insanely awesome test suite. Written in C, doesn’t cross compile nicely with the Go toolchain. Setting up build environments for all my target platforms is more pain than I need.


Native Go SQL-like data store. Lacking a few relevant SQL features mainly around indexes, has some issues (a weirdly hidden WAL file and no guarantee that a successful transaction commit will actually make it to disk).


Key-value store that underlies QL, so shares the limitations apart from not being SQL-like.


Uses the leveldb shared library, so no cross compilation and no single binary deployment.


Hasn’t implemented enumeration which I need from a kv store.


Native Go key-value store, requires roundtripping data through JSON. Clobbers large integers, fairly inefficient.


Native Go key-value store, stores each item as a separate file on disk…

Might Work

These are the ones I’m working on integrating with at the moment.


Native Go key-value store. Has some bugs and issues that are being sorted out, if it works out it looks nice.


Yet another leveldb implementation, this one in pure Go and looks promising. Still early in the integration work.

1 Like

I’ve selected leveldb, and it seems to be working out well. Hopefully this can be merged to master soon, and then needs some testing from other people…

There’s a bunch of stuff that will need to be a little bit rewritten to take advantage of the new backend as well, in order to gain the memory efficiency we need.

Will the index be stored on disk instead of RAM? For large shares the amount of ram used is too much.

Yep, that’s the point of the exercise. The current master branch does this, and syncthing currently (on my computers) uses about 160 MB to sync 500 GB of files to a few other nodes. This could probably be tweaked down further, but it’s better than the gigs and gigs it would need previously.