proxy.golang.org: '400 bad request' for pseudo-versions that refer to commits that also have canonical version tags

What version of Go are you using (go version)?

$ go version
go version go1.13beta1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOHOSTARCH="amd64"
GOHOSTOS="linux"

Ubuntu Disco Dingo, go1.13beta1 downloaded from the Golang site.

What did you do?

go get -d github.com/future-architect/vuls
cd ~/go/src/github.com/future-architect/vuls
  # see that it has a `go.mod` already, so ..
go install

What did you expect to see?

Either a successful build or some kind of build failure

What did you see instead?

A bad request from proxy.golang.org:

go: github.com/tomoyamachi/reg@v0.16.2-0.20190418055600-c6010b917a55: reading https://proxy.golang.org/github.com/tomoyamachi/reg/@v/v0.16.2-0.20190418055600-c6010b917a55.mod: 400 Bad Request

I repeated the command, to rule out transient glitch. I then ran GOPROXY=direct go install which continued past that point to get to a different inscrutable error but one which is more likely tied to the project itself. Limiting this issue to the observed proxy error.

[bcmills]

to get to a different inscrutable error but one which is more likely tied to the project itself

That suggests that the proxy probably should have instead returned a 404 or 410 to prompt the go command to fall back to direct.

CC @hyangah @katiehockman @heschik

[heschi]

If you visit the URL directly you'll see that it says "invalid version". That could be improved, but I don't know how much we should invest in it until #30748 is fixed; odds are nobody will see it anyway.

[heschi]

@bcmills: https://proxy.golang.org/github.com/tomoyamachi/reg/@v/v0.16.2-0.20190418055600-c6010b917a55.info is (for better or worse) a pointer to v0.16.1. Do you know why the go command is still reading v0.16.2-0.20190418055600-c6010b917a55.mod? This seems like a variation of the wrong .zip bug that was fixed in 1.13?

[bcmills]

@heschik, the validation rule in 1.13beta1 is that the base tag for the pseudo version must be on the commit itself or one of its ancestors, so v0.16.2-0.20190418055600-c6010b917a55.info is an allowed name for that commit.

I suppose we could tighten up that validation so that the base tag must be on some actual ancestor rather than the commit itself. I don't have a solid intuition one way or the other on that (cc @jayconrod).

[bcmills]

Hrm, that's odd. Using go1.13beta1, I get a different result from what @it-philpennock observed:

$ GO111MODULE=on go1.13beta1 get -d github.com/tomoyamachi/reg@v0.16.2-0.20190418055600-c6010b917a55
go: finding github.com/tomoyamachi/reg v0.16.2-0.20190418055600-c6010b917a55
go: finding github.com v0.16.2-0.20190418055600-c6010b917a55
go: finding github.com/tomoyamachi v0.16.2-0.20190418055600-c6010b917a55
go get github.com/tomoyamachi/reg@v0.16.2-0.20190418055600-c6010b917a55: github.com/tomoyamachi/reg@v0.16.2-0.20190418055600-c6010b917a55: proxy returned info for version v0.16.1 instead of requested version

nb: I was installing github.com/future-architect/vuls and tomoyamachi/reg is a dependency of that.

[bcmills]

Ah, here we go. I guess we try to download the .mod without bothering with the .info if it's already in the go.mod and go.sum files.

For completeness:

$ GO111MODULE=on go1.13beta1 get -d github.com/future-architect/vuls
go: finding github.com/future-architect/vuls v0.8.2
verifying github.com/genuinetools/reg@v0.16.1/go.mod: github.com/genuinetools/reg@v0.16.1/go.mod: reading https://sum.golang.org/lookup/github.com/genuinetools/reg@v0.16.1: 410 Gone

$ git clone https://github.com/future-architect/vuls
Cloning into 'vuls'...
remote: Enumerating objects: 24, done.
remote: Counting objects: 100% (24/24), done.
remote: Compressing objects: 100% (16/16), done.
remote: Total 5310 (delta 10), reused 16 (delta 8), pack-reused 5286
Receiving objects: 100% (5310/5310), 5.33 MiB | 27.58 MiB/s, done.
Resolving deltas: 100% (3679/3679), done.

$ cd vuls

vuls$ go1.13beta1 list all
go: github.com/tomoyamachi/reg@v0.16.2-0.20190418055600-c6010b917a55: reading https://proxy.golang.org/github.com/tomoyamachi/reg/@v/v0.16.2-0.20190418055600-c6010b917a55.mod: 400 Bad Request

[bcmills]

FWIW, there are other bogus versions mentioned in that same go.mod file:

vuls$ GOPROXY=direct go1.13beta1 list all
go: github.com/kotakanbe/goval-dictionary@v0.1.3-0.20190613053258-078b163b76ec: invalid pseudo-version: does not match version-control timestamp (2019-06-13T06:46:14Z)

[gopherbot]

Change https://golang.org/cl/184720 mentions this issue: cmd/go: tighten the check for pseudo-version base tags

[bcmills]

CL 184720 will tighten up the validation so that this pseudo-version is treated as actually-invalid.

However, there remains a bug in the proxy for valid pseudo-versions naming commits that have subsequently had canonical tags applied:

example.com$ go mod init example.com
go: creating new go.mod: module example.com

example.com$ go version
go version devel +fbde753a58 Tue Jul 2 13:52:21 2019 +0000 linux/amd64

example.com$ GOPROXY='' go get -d golang.org/x/text@v0.1.1-0.20171213102548-c4d099d611ac
go: finding golang.org/x/text v0.1.1-0.20171213102548-c4d099d611ac
go: finding golang.org v0.1.1-0.20171213102548-c4d099d611ac
go: finding golang.org/x v0.1.1-0.20171213102548-c4d099d611ac
go get golang.org/x/text@v0.1.1-0.20171213102548-c4d099d611ac: golang.org/x/text@v0.1.1-0.20171213102548-c4d099d611ac: proxy returned info for version v0.2.0 instead of requested version

example.com$ GOPROXY=direct go get -d golang.org/x/text@v0.1.1-0.20171213102548-c4d099d611ac
go: finding golang.org/x/text v0.1.1-0.20171213102548-c4d099d611ac
verifying golang.org/x/text@v0.1.1-0.20171213102548-c4d099d611ac/go.mod: golang.org/x/text@v0.1.1-0.20171213102548-c4d099d611ac/go.mod: reading https://sum.golang.org/lookup/golang.org/x/text@v0.1.1-0.20171213102548-c4d099d611ac: 410 Gone

example.com$ GOPROXY=direct GOSUMDB=off go get -d golang.org/x/text@v0.1.1-0.20171213102548-c4d099d61
1ac
go: downloading golang.org/x/text v0.1.1-0.20171213102548-c4d099d611ac
go: extracting golang.org/x/text v0.1.1-0.20171213102548-c4d099d611ac