Changes related to proto3 getting reverted

With the update to proto 1.2.0 in https://github.com/syncthing/syncthing/pull/5452 a proto3 option has been added in quite few places, some file headers removed, … When I run go run build.go proto on current master (i.e. with that PR) those changes get reverted. The package I am using that provides protoc is at v3.6.1, which is up-to-date. Anything else I could be missing?

go install the protoc-gen-gogofast from v1.2.0 (in repos/protobuf for example), I think that’s the thing.

That works for me anyway, on both my computers now, with protoc 3.6.1 like you.

1 Like

Ok that worked - thanks.

Shouldn’t this be handled by build.go, in additional to downloading the correct repo which it already does?

It can’t really handle all of it (installing protoc etc)… But I guess it could install the gen-gogofast binary somewhere and tweak $PATH to point to it, etc.

For complex builds and specific tooling dependencies Bazel could be an option, only you introduce a fearly new open-sourced software component into the project but could be a nice experiment to have reproducible builds including same tooling.

I do hope that

was not a typo. I see it as a wonderfully useful expression.:wink:

1 Like

Running go run build.go proto and then go generate for the lib/db/ directory still produces an enormous diff, including some proto3 flags getting removed.

What is the correct way to build modified protobuf definitions in Syncthing? What tools (and versions) are actually required?

I noticed on my Ubuntu 18.04 system, protoc --version is still 3.0.0, even after installing protoc-gen-gogofast as mentioned above. Go 1.12 is installed via Snap, so how would I get a newer protoc?

I see the complexity in having to install all these really young and evolving development tools, but isn’t it generally bad practice to keep generated files under version control?

You need protobuf version 3.something installed. I don’t think the version is particularly critical. On my machine,

$ protoc --version
libprotoc 3.7.1

Ensure that ~/go/bin (or the equivalent in $GOPATH if you’ve set that explicitly) is early in your path, because that’s where the right version of protoc-gen-gogofast will get installed. Run go run build.go proto. This installs the expected version and runs go generate as needed.

Do not run go generate manually; it’s not necessary, and might do the wrong thing depending on what you have in your $PATH.

In general, sure. In this specific case, definitely not.

I guess that was the main mistake then. Modifying $PATH to include ~/go/bin worked, with Ubuntu’s protoc version 3.0.0.

Would be better if the build script allowed to specify a custom search path for these tools. Having users adjust their $PATH for development tools that are installed from within the Syncthing repo seems wrong. Could it be made to install locally under lib/ or something like tools/?

Now the diff looks much cleaner, though this stuns me in go.mod:

 	github.com/d4l3k/messagediff v1.2.1
 	github.com/flynn-archive/go-shlex v0.0.0-20150515145356-3f9db97f8568
 	github.com/gobwas/glob v0.0.0-20170212200151-51eb1ee00b6d
-	github.com/gogo/protobuf v1.2.1
+	github.com/gogo/protobuf v1.2.0
 	github.com/golang/groupcache v0.0.0-20171101203131-84a468cf14b4
 	github.com/golang/snappy v0.0.0-20170215233205-553a64147049 // indirect
 	github.com/jackpal/gateway v0.0.0-20161225004348-5795ac81146e
 	github.com/kballard/go-shellquote v0.0.0-20170619183022-cd60e84ee657
+	github.com/kisielk/errcheck v1.1.0 // indirect
 	github.com/kr/pretty v0.1.0 // indirect
 	github.com/lib/pq v1.1.1
 	github.com/mattn/go-isatty v0.0.7

Yeah, that’s the build script doing something unexpected. The whole build.go proto thing is a bit nasty.