Bad hash on quic-go

#1

If you try to build Syncthing from scratch with no module cache in place you get the following:

jb@kvin:~ $ ./build.sh
go: downloading github.com/lib/pq v1.2.0
go: downloading github.com/prometheus/client_golang v1.1.0
...
verifying github.com/lucas-clemente/quic-go@v0.12.0: checksum mismatch
	downloaded: h1:TRbvZ6F++sofeGbh+Z2IIyIOhl8KyGnYuA06g2yrHdI=
	go.sum:     h1:dYHUyB50gEQlK3KqytmNySzuyzAcaQ3iuI2ZReAfVrE=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

It seems our module proxy (Athens) did something wrong with this package or the tag moved since it saw it first. It works on the build servers since they have the old/broken package in their module cache and/or use that proxy.

I’m going to have to fix this, and then I suspect the build will break for everyone who has it working today. Yay. :expressionless: The solution at that point seems to be to blow away the module cache (~/go and ~/.cache/go-build) and use the default GOPROXY.

Currently it works with GOPROXY=https://build.syncthing.net/athens as it serves the same copy that has been hashed previously.

#2

Did you already change something on athens? The android builder that uses that passed this afternoon and fails now due to quic-go.

I just tried the following with go1.13:

  1. Remove go cache and $GOPATH/pkg/mod/…/quic-go@v1.12.0
  2. Remove quic-go from our go.sum.
  3. Build.

Result:

verifying github.com/lucas-clemente/quic-go@v0.12.0: checksum mismatch
        downloaded: h1:dYHUyB50gEQlK3KqytmNySzuyzAcaQ3iuI2ZReAfVrE=
        sum.golang.org: h1:TRbvZ6F++sofeGbh+Z2IIyIOhl8KyGnYuA06g2yrHdI=

This is fixed by setting GOSUMBD=off.

So this seems like a genuine problem for upstream that will likely require a 0.12.1 release. I’ll open an issue.

1 Like
#3

I didn’t do anything with the proxy yet, no.

#4

Weird.

If I remove all the cache but leave our go.sum in place, I don’t have any problems. So could it be that the checksum in our go.sum is compatible with the current quic-go, but athens (and google) have an old version and thus fail if there’s no cache?

#5

it’s all very mysterious :grimacing: :pensive:

#6

Turns out I messed something up when “reproducing” - there is no issue if I remove our go.sum. So the problem is in our go.sum (and athens?).

We probably need to change our go.sum to contain the “new” hash and release a new RC with a note that one needs to clear the mod cache if there’s a checksum error on quic-go.

#7

I’ve done the deed;

  • Corrected the hash in go.sum
  • Killed the Athens proxy and told the build server not to use it
  • Cleared the module cache on all the builders

It now builds using the default Go proxy and sumdb stuff. I’m not going to revive the Athens proxy as it seems to cause issues and isn’t strictly necessary.

I don’t think this warrants a new RC on its own. I’ll make sure it ends up in the release though, if we don’t make any further RCs along the way.

#8

The android release hasn’t happened, so that will be a dev not candidate build without a new rc. I don’t think there’s practical implications except for the displayed version string, so that should be fine.

#9

Well I can tag an rc.2 if you like. Or you can.

#10

Having read https://docs.syncthing.net/dev/release-creation.html I decided that a tag+1 build for the android rc is just fine :slight_smile:

#12

It’s just a small subset of the whole rigmarole to push a new tag :slight_smile: But sure.

#13

Sure, but I wasn’t sure what effects just pushing a tag has on the whole rigmarole of the next tag. Probably none, but still :slight_smile:

1 Like
#14

This is driving me fucking nuts. I have the same problem. Nuked $GOPATH/pkg, removed the code I have for the package, ran go clean -modcache, still same issue.

#15

I also had this issue last week (6 days ago, to be exact) on a test machine which had a clean copy of master. I simply removed the quic-go entry from go.sum, afterwards everything worked.

#16

yeah but that changes the hash to the incorrect one

#17

Are you on Go 1.13? I think 1.12 still writes the bad hash. Or if you use a proxy that uses the module packages from Go 1.12…

In effect I think you can only build with 1.13 now, or by disabling the sums. (GONOSUMDB=1 iirc)

#18

I have the same issue. After some googling I found another project (go-micro) that also use quic-go where the same issue occur. See discussion in: