Looking at the protocol description and the software in general I see that the protocol is a kind of generic thing for creating distributed systems that can share different things (while organized in a file-like structure these do not necessarily be files). The application itself is specific to file sharing. What about splitting the protocol implementation into a separate library?
There are already implementations in different languages (java, swift, I am working on a PHP implementation). The lib could be used for multiple different implementations of distributed applications while the application stays specific to file sharing.
I have no idea what @aral is creating with ind.ie and I’m afraid we do not get much details about this until Nov. 8 2014, but as it is based on this tool I assume it most likely does not use the frontend and file sharing part but only the protocol to sync between instances.
This may also be a solution to the naming thing, letting Syncthing be the app and pulse be the library?
The current implementation language (Go) is awesome for many things, but unfortunately very ill suited for creating a loadable library. I think a C or Java implementation would be a better base to build on. (No idea about Swift really but I suspect it may have similar issues to Go, seeing as it has garbage collection and thus a runtime of its own.)
Well, my main point was not that it should be a loadable library, even splitting out the Go implementation as a separate package would allow using it in other Go projects. The main idea here is to have a reference implementation of the protocol in a separate package that defines the scope of this package regarding included features and the protocol specs.
This would allow porting it to many different languages so that many programmers can build distributed systems on it. As a separate library it can get broader usage not limited to file sharing.
Well, in that sense this is already done; the package is github.com/syncthing/syncthing/internal/protocol. It’s under the internal namespace (because we’re the only consumer), but there’s nothing to stop anyone from pulling it out.
Cool, did not notice that. Its a bit hard to figure out how everything works together. I have never worked with Go before and there are near to no comments in your code
Guilty as charged… That’s old code by now, written in proof of concept frenzy (although it works quite well). Many newer contributions are clearer.
This is a really interesting topic. I think this idea should be reconsidered in the near future.
The SyncthingFUSE app uses many packages of the main syncthing repo and works like a standalone application, with its own data and configuration, that basically uses the syncthing protocol to talk to other people running syncthing.
It’s a use case for the idea of more explicitly separating code for the app and code for the app internals, if writing this kind of program is something we would like to encourage.
This post came up while I was searching for a way to use Syncthing from Java or other tools.
I wanted to ask what is the status of this.
Is it possible to do that?
How easy would it be for a developer not familiar with go - but familiar with other languages and building packages in general?
It seems that there is some work to publish go package as a library:
I could open an issue for this to be tracked.
I think it is useful to be able to call the reference implementation from other languages for many reasons, one of them being improved compatibility between implementations as this would allow better testing IMO.
Yes, Syncthing Tray allows building Syncthing itself as a library. It provides a simple C++ interface but this is limited to starting the Syncthing service, invoking CLI commands and a few other functions. The logging is not a problem; you can register a logging callback, e.g. Syncthing Tray uses this to show stdout/stderr in the UI.
I suppose that’s still not really what @ieugen had in mind, though. You cannot use my approach for building Syncthing to produce a reusabe Go package. I have actually tried this once. The idea was to build Syncthing as a reuable Go package and then have the C-bindings as a separate Go package depending on it. However, this approach failed because I was unable to fiddle the asset generation into Go’s build process. So this part is currently handled at CMake-level (but using CMake doesn’t seem to be a possibility for a Go package).
Another problem (not for Syncthing Tray but for further use cases) is likely the absence of a stable and easy to use interface for more detailed use cases than just starting the whole service (which you can also have by just starting it as separate process). For instance, one might want a function to just fetch/push a certain file from/to a certain device. Or one might even want the full service but control it directly via functions (instead of having to use HTTP requests).
A general remark: Creating bindings from one high-level language (like Go) to another high-level language (like Java) is always challenging. So likely one would end up writing C-bindings and then wrapping those bindings to use them in other languages. Even using the headers generated by cgo directly from C might not be desirable as they still contain language-specific things like GoInt64. (Note that cgo command - cmd/cgo - Go Packages actually mentions some Java-specifics, though.)
I was just copying the term from @ieugen’s message. I meant a “Go Module” specifically. So Syncthing would be a separate Go Module and my C-bindings would be a Go module depending on it.
That’s old code by now, written in proof of concept frenzy (although it works quite well). Many newer contributions are clearer.