cmd/go: GOFLAGS=-mod=vendor affects result of `go list -m all`

[pohly]

What version of Go are you using (go version)?

$ go version
go version go1.13.4 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/pohly/.cache/go-build"
GOENV="/home/pohly/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/nvme/gopath"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/nvme/gopath/go1.13.4"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/nvme/gopath/go1.13.4/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/nvme/gopath/src/github.com/intel/pmem-csi/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build014465635=/tmp/go-build -gno-record-gcc-switches"

What did you do?

• check out https://github.com/pohly/pmem-csi/tree/go-vendor-test-2019-11-14 (tag: go-vendor-test-2019-11-14)
• run diff <(GOFLAGS=-mod=vendor go list -f {{.Path}} -m all) <(go list -f {{.Path}} -m all)

What did you expect to see?

There should be no difference or, if that for some reason isn't feasible, at least all modules that are in the vendor directory should be listed.

What did you see instead?

The command without GOFLAGS=-mod=vendor reports a lot more modules. For example:

$ diff <(GOFLAGS=-mod=vendor go list -f {{.Path}} -m all) <(go list -f {{.Path}} -m all)
1a2,18
> bitbucket.org/bertimus9/systemstat
> cloud.google.com/go
> github.com/Azure/azure-sdk-for-go
> github.com/Azure/go-ansiterm
> github.com/Azure/go-autorest/autorest
> github.com/Azure/go-autorest/autorest/adal
> github.com/Azure/go-autorest/autorest/date
> github.com/Azure/go-autorest/autorest/mocks
> github.com/Azure/go-autorest/autorest/to
> github.com/Azure/go-autorest/autorest/validation
> github.com/Azure/go-autorest/logger
> github.com/Azure/go-autorest/tracing
> github.com/BurntSushi/toml
> github.com/BurntSushi/xgb
> github.com/GoogleCloudPlatform/k8s-cloud-provider
> github.com/JeffAshton/win_pdh
> github.com/MakeNowJust/heredoc
...
> k8s.io/kubectl
...

Many of these extra modules are not actually used and thus not present in vendor (for example, bitbucket.org/bertimus9/systemstat).

But some that aren't reported by the first command are used (for example, k8s.io/kubectl). That's more problematic because additional unused modules can be ignored, but if the module isn't even reported, it will be missed when analyzing dependencies.

[andybons]

@bcmills @jayconrod

[bcmills]

There should be no difference or, if that for some reason isn't feasible, at least all modules that are in the vendor directory should be listed.

It is, unfortunately, not possible for go list to compute a correct, complete version of all using the information in the vendor directory. In Go 1.14, go list -m will explicitly reject wildcard patterns when -mod=vendor is in use. (That is already implemented.)

go list -m will continue to be able to list any module that provides an actual package in the vendor directory.

[bcmills]

Addressed by CL 199821 and CL 203138. (See also #33848.)

[jayconrod]

In general, go list -m all cannot give complete results with -mod=vendor. Listing all modules requires loading the module graph, and we don't save enough information in go.mod and vendor/modules.txt to be able to do that.

In Go 1.14, the following message is printed:

go list -m: can't compute 'all' using the vendor directory
	(Use -mod=mod or -mod=readonly to bypass.)

@bcmills Maybe this is a documentation issue? go help list should probably explain the interaction when describing -m.

[bcmills]

@jayconrod, it may be that we should update the documentation, although I doubt that anyone will read it ahead of time and I think the error message mostly speaks for itself.