Should we move the smaller repos back into the main one?

Development
1

I’m considering scrapping the “external” repos protocol, discosrv, relaysrv, and relaypoolsrv. Maybe also syncthing-cli. These would be adopted back into the main repo. Why would I want such a thing? Because it keeps things in sync, without annoying vendoring steps on our own code.

I had to rebuild the discosrv for v0.11 yesterday, because I mislaid the binary. To do that, I had to

  1. Check out the old commit that I wanted on discosrv.
  2. Fail to build it. Realize that I had to find the corresponding commit in protocol and check that out.
  3. Fail to build it. Realize that I had to find the corresponding commit in syncthing and check that out.
  4. Fail to build it anyway, as it wanted stuff in internal/ which is no longer supported and was anyway a bad idea…
  5. Build it with an older compiler that accepted the internal/ prefix.

Did I actually get the same code as I had running previously, or did I select the wrong commit somewhere along the way? Fuck if I know… In the end I ended up stopping and rolling back the VM where the binary lived from a snapshot instead.

So anyway, we could tag stuff in protocol whenever we release a discosrv, or we could vendor it into that repo… But it’s our own code, and it evolves in lockstep. Moving it all into the same repo would have these advantages, as I see it:

  • The above situation doesn’t come up.
  • We catch build breakage immediately. (Currently, I can hack on the protocol and fix syncthing for it, and not really notice I broke discosrv in the meantime.)
  • We avoid having to vendor our own stuff.
  • We get all the binaries we want, built in one fell swoop…
  • Those binaries all get version tagged by default.

There are some disadvantages, as always:

  • We’ll lose the history of those repos when importing them. Well, the history will still be visible in those repos, but won’t be imported into the main repo.
  • It gets more cumbersome to use just the library in question, for non-syncthing consumers. I think the only place where this is relevant is the protocol package, and if there are any actual users of that out there I can commit to maintaining it at it’s current location, as a mirror of that inside the syncthing repo.
  • The things have different licenses. This doesn’t differ from now really though, as the code gets included with it’s own license anyway by the vendoring mechanism.
  • We lose one level of access control, as you could today theoretically be a contributor with push access to discosrv but not syncthing. I don’t see this as an issue at all - you’re welcome to more push access.
2

Other options:

  1. Use git submodules to keep the different components’ versions in lockstep
  2. Use git-subtree to split components out into separate repositories on a continual basis if required
3

Is a possibility. We’d get syncthing including protocol as a submodule, and for example the relayserver including both syncthing and protocol as submodules. It’s not super pretty in my opinion, but it is better than now.

4

It is quite easy to import everything with keeping history:

$ git subtree add --prefix=DIRECTORY github.com/syncthing/PROJECT master
5

Huh. That did something weirdly interesting for protocol, as we already had that history in the main repo… Sort of…

Ah, no, it “just” added the history as you said, which happens in parallell with the stuff we already had there. Neat. Removing that disadvantage.

6

I am up for anything that allows me to use less of godep.

I did prefer to have everything in a single place, as there was less git juggling to do too.

3 Likes
7

So hypothetically this is what that could look like:

https://github.com/calmh/syncthing/tree/include2

This adds (using the subtree method):

protocol      -> lib/protocol
relaysrv      -> cmd/relaysrv
relaypoolsrv  -> cmd/relaypoolsrv
discosrv      -> cmd/discosrv
syncthing-cli -> cmd/syncthing-cli

And then moves:

cmd/relaypoolsrv/client   -> lib/relay/client
cmd/relaypoolsrv/protocol -> lib/relay/protocol

And rewrites the various import paths. We gained a few new vendored dependencies (database stuff for the discosrv and so on).

We’d then work as usual, but maintain protocol as an external mirror of syncthing/lib/protocol somehow…

8

Not sure how we’d handle “releases” of the discovery and relay servers, I mean where the binaries are made available and so on. In their own repos, that only contain a README pointing to syncthing? Just link to whatever’s the latest on the build server?

9

Yeah. The most important binary is syncthing. I think the relay- and discoserverstuff could be moved somewhere else. Distributions will build their packages by themselves anyway… So, personally I don’t really think that prebuild binaries of the server stuff are that important.

syncthing-cli should imho be bundled with syncthing.

10

I think it’s very useful to host them, but sticking them on the build server is fine IMO

11

Yeah; imo there is no need to build a separate download page or something for that stuff.

12

3 posts were split to a new topic: Move protocol specs to docs.syncthing.net

14

So I feel number of tweaks on relaysrv and other standalone tools all are relatively small, and they are quite independant. On the other hand, protocol is a core part, and a bitch to work with being on a separate repo.

My proposal would be to move protocol back into lib, move relay protocol and client into lib/relay, and just fix broken *srv jobs as they break, or add vendoring to *srv (which should be painless as we don’t expect them to change much)

cli is also packaged by some people, so perhaps should stay where it is.

15

That’s also an idea, yes.

16

I think I’ll go with that. That’ll be one change on the syncthing repo, then vendoring on the “outliers”. So ironically, more godep (but maybe used less often).

https://github.com/syncthing/relaypoolsrv/pull/1

That’s a lot of my face, right there.