Hosted syncthing at rsync.net - technical issues ...

We (rsync.net) are interested in supporting syncthing users - I am interested in this personally and a fair number of customers have asked about it.

Our platform is ssh only and is accessible only over port 22. There are no services or daemons running - only OpenSSH. This works just fine for tools ‘borg’ or ‘restic’ - either the user calls the remote binary over ssh:

ssh user@rsync borg -xyz blah blah

Or, the tool itself supports SFTP as an endpoint (like restic).

I am aware that syncthing does not support SFTP as a transport and I think that syncthing requires a server-side binary in place …

But does it need to run as a daemon/server ?

Can it be executed, remotely, over SSH:

ssh user@rsync syncthing -xyz blah blah

I am wondering if users could have the hosted syncthing binary at rsync.net behave as the primary which would push and pull data from other sources (laptops, servers, etc.) out in the world ?

Or perhaps there is not a decent way to make this work ?

Comments appreciated. Thank you.

1 Like

I think the reason why your customers talk about syncthing is because it’s continuous, so I don’t think one-shot ssh server syncthing -xyz would give them what they are looking for, otherwise how does it differ from borg/restic/rsync?

Yes, syncthing is a daemon as implemented, I guess you could have the binary stored that the users could launch, it does it’s thing and then somehow gets terminated, but as I said I don’t see how this is different from what you have now with other tools.

Syncthing needs a database, it needs a config, it’s hashing is expensive, it sends out multicast/broadcast to local network, connects to discovery servers/relays, a lot of stuff you don’t need. Also, it requires to be always running and a continuous connection.

Sure you can probably silence parts of that and perhaps hold it sideways to make it work, but to be honest, I think your users would probably be just as happy with rsync --watch or something along those lines which effectively gives them the “continuous” aspect of syncthing via other tools.

Thank you - that’s interesting.

I wonder if it would make more sense to mount the rsync.net account (or any sftp login anywhere, I guess) using sshfs (FUSE) and then point your local syncthing operation to that local (sshfs) mountpoint ?

This kind of thing used to be a bad idea but the efficiency and granularity of sshfs these days is now fine enough that you can do things like this and have it work reasonably well …

I did a quick search and it looks like a recipe that others are using:

Comments ? I might respond to some of the customers and suggest they start with this recipe …

That sounds doable.

Syncthing itself uses filesystem providers that abstract away the underlying filesystem. Currently we only have implementations for a normal POSIX filesystem and a small in-memory fake filesystem for tests. With some effort it should be possible to write a filesystem implementation that speaks directly to an S3-like API or similar, which might suit you better. Such an implementation would likely be welcomed.

This doesn’t really help in the context of rsync.net users tho, because we don’t support syncing between local folders, so you’d still have to run two instances on the same machine, one mounting local filesystem, another one mounting whatever rsync.net is exposing, and then setting up sync between the two, and let them do syncthing protocol over tls over tcp etc, just to copy files around between to locations on the same machine.

If you mounted sshfs, you might as well just run unison or something for syncing between to local paths.

1 Like

Syncthing itself uses filesystem providers that abstract away the underlying filesystem. Currently we only have implementations for a normal POSIX filesystem and a small in-memory fake filesystem for tests. With some effort it should be possible to write a filesystem implementation that speaks directly to an S3-like API or similar, which might suit you better. Such an implementation would likely be welcomed.

Wouldn’t this be similar to running two separate syncthing instances watching the same directory in a single machine? I imagine there could be some conflicts.

One possible solution would be to have daemon in the user’s machine that pretends to be syncthing (listening on custom TCP/UDP ports). The user’s main syncthing instance would then be configured to talk to the fake syncthing like it was a separate node. When the fake syncthing receives a TCP/UDP connections, it would open an ssh connection to rsyncnet and proxy the connections.

On rsyncnet side there could be a syncthing manager daemon that is spawned on demand, and spawns syncthing child while also forwarding connections to it. This daemon would stop (and consequently kill syncthing) when the last ssh connection is closed. The purpose of having a daemon is to allow multiple syncthing nodes to talk to the rsyncnet node simultaneously, if syncthing was spawned for every ssh connection there could be multiple syncthing instances working on the same set of shared directories (which I imagine is a similar problem to the sshfs idea).

There are multiple possible solutions for sure. This is rsync.net’s commercial service, so their problem to solve, really. :slight_smile: I would hope that some successful integration somehow results in a contribution back to the community.