Why does syncthing use such a strange building system

Development
21

Well, I’m building in a clean jail with nothing but the OS and go and the source (fetched tar for latest master from github). When I build, I see:

`root@10amd64-default:/tmp/work/syncthing-master # env GOPATH=/tmp/work/syncthing-master/Godeps/_workspace:/tmp/work/syncthing-master go build ./cmd/syncthing cmd/syncthing/gui.go:24:2: cannot find package “github.com/syncthing/syncthing/internal/auto” in any of: /usr/local/go/src/pkg/github.com/syncthing/syncthing/internal/auto (from $GOROOT) /tmp/work/syncthing-master/Godeps/_workspace/src/github.com/syncthing/syncthing/internal/auto (from $GOPATH) /tmp/work/syncthing-master/src/github.com/syncthing/syncthing/internal/auto cmd/syncthing/gui.go:25:2: cannot find package “github.com/syncthing/syncthing/internal/config” in any of: /usr/local/go/src/pkg/github.com/syncthing/syncthing/internal/config (from $GOROOT) /tmp/work/syncthing-master/Godeps/_workspace/src/github.com/syncthing/syncthing/internal/config (from $GOPATH) /tmp/work/syncthing-master/src/github.com/syncthing/syncthing/internal/config cmd/syncthing/main.go:31:2: cannot find package “github.com/syncthing/syncthing/internal/discover” in any of: /usr/local/go/src/pkg/github.com/syncthing/syncthing/internal/discover (from $GOROOT) /tmp/work/syncthing-master/Godeps/_workspace/src/github.com/syncthing/syncthing/internal/discover (from $GOPATH) /tmp/work/syncthing-master/src/github.com/syncthing/syncthing/internal/discover cmd/syncthing/gui.go:26:2: cannot find package “github.com/syncthing/syncthing/internal/events” in any of: /usr/local/go/src/pkg/github.com/syncthing/syncthing/internal/events (from $GOROOT) /tmp/work/syncthing-master/Godeps/_workspace/src/github.com/syncthing/syncthing/internal/events (from $GOPATH) /tmp/work/syncthing-master/src/github.com/syncthing/syncthing/internal/events cmd/syncthing/main.go:33:2: cannot find package “github.com/syncthing/syncthing/internal/files” in any of: /usr/local/go/src/pkg/github.com/syncthing/syncthing/internal/files (from $GOROOT) /tmp/work/syncthing-master/Godeps/_workspace/src/github.com/syncthing/syncthing/internal/files (from $GOPATH) /tmp/work/syncthing-master/src/github.com/syncthing/syncthing/internal/files cmd/syncthing/gui.go:27:2: cannot find package “github.com/syncthing/syncthing/internal/logger” in any of: /usr/local/go/src/pkg/github.com/syncthing/syncthing/internal/logger (from $GOROOT) /tmp/work/syncthing-master/Godeps/_workspace/src/github.com/syncthing/syncthing/internal/logger (from $GOPATH) /tmp/work/syncthing-master/src/github.com/syncthing/syncthing/internal/logger cmd/syncthing/gui.go:28:2: cannot find package “github.com/syncthing/syncthing/internal/model” in any of: /usr/local/go/src/pkg/github.com/syncthing/syncthing/internal/model (from $GOROOT) /tmp/work/syncthing-master/Godeps/_workspace/src/github.com/syncthing/syncthing/internal/model (from $GOPATH) /tmp/work/syncthing-master/src/github.com/syncthing/syncthing/internal/model cmd/syncthing/gui_csrf.go:19:2: cannot find package “github.com/syncthing/syncthing/internal/osutil” in any of: /usr/local/go/src/pkg/github.com/syncthing/syncthing/internal/osutil (from $GOROOT) /tmp/work/syncthing-master/Godeps/_workspace/src/github.com/syncthing/syncthing/internal/osutil (from $GOPATH) /tmp/work/syncthing-master/src/github.com/syncthing/syncthing/internal/osutil cmd/syncthing/gui.go:29:2: cannot find package “github.com/syncthing/syncthing/internal/protocol” in any of: /usr/local/go/src/pkg/github.com/syncthing/syncthing/internal/protocol (from $GOROOT) /tmp/work/syncthing-master/Godeps/_workspace/src/github.com/syncthing/syncthing/internal/protocol (from $GOPATH) /tmp/work/syncthing-master/src/github.com/syncthing/syncthing/internal/protocol cmd/syncthing/gui.go:30:2: cannot find package “github.com/syncthing/syncthing/internal/upgrade” in any of: /usr/local/go/src/pkg/github.com/syncthing/syncthing/internal/upgrade (from $GOROOT) /tmp/work/syncthing-master/Godeps/_workspace/src/github.com/syncthing/syncthing/internal/upgrade (from $GOPATH) /tmp/work/syncthing-master/src/github.com/syncthing/syncthing/internal/upgrade cmd/syncthing/main.go:38:2: cannot find package “github.com/syncthing/syncthing/internal/upnp” in any of: /usr/local/go/src/pkg/github.com/syncthing/syncthing/internal/upnp (from $GOROOT) /tmp/work/syncthing-master/Godeps/_workspace/src/github.com/syncthing/syncthing/internal/upnp (from $GOPATH) /tmp/work/syncthing-master/src/github.com/syncthing/syncthing/internal/upnp root@10amd64-default:/tmp/work/syncthing-master #

`

I guess it wants all source to be under src... ? I mean, I could just copy it over, but from what you described that shouldn’t be necessary.

22

Right. So the path to the source must be $wherever/src/github.com/syncthing/syncthing. I get that that’s annoying and unusual, but that’s how the Go build system works and expects things to be set up. The $wherever should be your $GOPATH, but the build script will try to figure it out if it isn’t. You shouldn’t play any tricks with $GOPATH to point at the included dependencies, because the build script does that on it’s own.

So the minimal build setup ought to be something like;

export GOPATH=/tmp/foo
# extract the source repo in /tmp/foo/src/github.com/syncthing/syncthing, somehow
cd /tmp/foo/src/github.com/syncthing/syncthing
go run build.go
23

Oh, and the master branch just landed some fairly big changes. There’s a v0.9 branch without those that’ll probably produce at least one or two more v0.9 releases, might want to track that one for the time being.

24

Oh, OK. Odd, but OK. And now that that works I realize that you were proposing to remove the requirement for git, but haven’t done so yet, which isn’t a big deal. In the mean time I can work on an update for the discovery server and cli. :slight_smile:

25

I usually just grab the release tar, so will do. Thanks!

26

Right, yes. The git requirement will go away, it just hasn’t yet. :wink:

27

Great, no worries. So a few more questions, if I may…

Would you be willing to make the same build changes to the discovery server and cli?

Is there any point in building/shipping the additional commands in the cmd subdir?

We normally strip our binaries before shipping them, but I notice you don’t. Is there any harm in stripping?

Is there any harm in using the newer go 1.3.2?

Edit: Oh and releases of the discovery server and cli that matched the other releases would be nice, if that’s possible.

Perhaps I should open a new thread? But these are somewhat build related.

28

Would you be willing to make the same build changes to the discovery server and cli?

Yeah, in the long run. I guess you mean with respect to them sucking in sources from the internet, as the build process is otherwise just a straight “go build”?

Is there any point in building/shipping the additional commands in the cmd subdir?

Nah

We normally strip our binaries before shipping them, but I notice you don’t. Is there any harm in stripping?

Yes, Go doesn’t like that at all. The build script already gets as close as possible with the -w ldflag that skips a bunch of debug info.

Is there any harm in using the newer go 1.3.2?

Should be perfectly fine, but I’ve been holding off since I saw complaints about test failures on Windows.

Edit: Oh and releases of the discovery server and cli that matched the other releases would be nice, if that’s possible.

Yeah, so the discovery server should update very seldom. At most, when the “major” version changes (i.e. 0.9 -> 0.10, before we hit 1.0), otherwise it’s compatible. And very niche, that is most users won’t need it at all.

The CLI, yeah, I guess. (cc @AudriusButkevicius etc).

29

So in regards to CLI, I simply have no clue how godeps works, hence I put the code in a state where I can build it, push it, and rely on jenkins on producing artefacts which most of the people can use.

You are more than welcome to produce a build script which builds on all platforms and manages the deps at the same time, though from previous experience with syncthing, I know that godeps was complaining about dirty repo straight after the pull on Windows (which is sadly what I use at home). Perhaps something todo with EOLs as @calmh suggested.

30

I think a git config --global core.autocrlf input and then re-checking-out the repos in question ought to solve that. That gives you unix line endings on disk, but any real editor should be able to handle that just fine.

31

Is this still planned? When might it happen? :smile:

32

It has. You need to pass -version v1.2.3 to build.go to set the actual version being built.

33