cmd/go: 1.17 `go mod tidy` produces go.mod files that cannot be used with `go mod graph` to understand the source of dependencies

[liggitt]

What version of Go are you using (go version)?

$ go version
go version go1.17rc2 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""

GOARCH="amd64"

GOBIN=""

GOCACHE="/Users/liggitt/Library/Caches/go-build"

GOENV="/Users/liggitt/Library/Application Support/go/env"

GOEXE=""

GOEXPERIMENT=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="darwin"

GOINSECURE=""

GOMODCACHE="/Users/liggitt/go/pkg/mod"

GONOPROXY=""

GONOSUMDB=""

GOOS="darwin"

GOPATH="/Users/liggitt/go"

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/Users/liggitt/.gvm/gos/go1.17rc2"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/Users/liggitt/.gvm/gos/go1.17rc2/pkg/tool/darwin_amd64"

GOVCS=""

GOVERSION="go1.17rc2"

GCCGO="gccgo"

AR="ar"

CC="clang"

CXX="clang++"

CGO_ENABLED="1"

GOMOD="/Users/liggitt/go/src/k8s.io/kubernetes/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/7f/9xt_73f12xlby0w362rgk0s400kjgb/T/go-build1198370725=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

hoisted from discussion at #46366 (comment)

Tidied a module with direct and indirect dependencies and ran through dependency stats checking tools based on go mod graph.

go mod init example

echo '
package main

import (
  "fmt"
  "github.com/liggitt/tidy-117/b"
)

func main() {
  fmt.Println(b.Do())
}
' > main.go

go mod tidy

go mod graph

What did you expect to see?

Useful module graph output.

go1.16 produces this:

example github.com/liggitt/tidy-117/b@v1.0.0
github.com/liggitt/tidy-117/b@v1.0.0 github.com/liggitt/tidy-117/c@v1.0.0

What did you see instead?

go1.17rc2 produces this:

example github.com/liggitt/tidy-117/b@v1.0.0
example github.com/liggitt/tidy-117/c@v1.0.0
github.com/liggitt/tidy-117/b@v1.0.0 github.com/liggitt/tidy-117/c@v1.0.0

That shows an edge from example → github.com/liggitt/tidy-117/c which does not actually exist... it is only transitive via github.com/liggitt/tidy-117/b.

go mod graph is currently useful for understanding subtrees of tidied dependencies, and piping to graph visualization tools like dot to find candidates for reducing dependencies.

The 1.17 changes to go mod tidy add edges that make go mod graph unsuitable for this.

[liggitt]

cc @bcmills @jayconrod @dims

[bcmills]

Tidied a module with direct and indirect dependencies and ran through dependency stats checking tools based on go mod graph.
…
The 1.17 changes to go mod tidy add edges that make go mod graph unsuitable for this.

I don't think go mod graph was ever really suitable for computing direct vs. indirect dependency stats. Its main use-case is to help you figure out why a specific version is selected, not whether or how that version is relevant to the packages within your module.

The dependency edges added for go 1.17 modules could also be present in other modules any time a transitive dependency is upgraded above the version that would otherwise be selected. So the assumption that “explicit” == “direct” was already quite fragile (it did not necessarily hold, although it may have happened to hold for many projects in practice).

[bcmills]

The tools that we have today for analyzing the reason a dependency is needed are go mod why (and go mod why -m) and go list.

There are also some third-party tools that build on top of those tools to provide more structure and support more detailed queries. (For example, some folks on the Gophers Slack this morning were recommending goda. I haven't used it myself but it seems to support a pretty reasonable query language.)

[liggitt]

The dependency edges added for go 1.17 modules could also be present in other modules any time a transitive dependency is upgraded above the version that would otherwise be selected.

Understood, but in that scenario, the module had an explicit opinion about the transitive dep, so representing an edge made sense.

I think it's unfortunate that the graph representation we previously got for free using go mod tidy and go mod graph we now have to reconstruct from the output of go list

[liggitt]

The tools that we have today for analyzing the reason a dependency is needed are go mod why (and go mod why -m) and go list.

Working on switching to use go list in kubernetes/kubernetes#104461 (comment), it looks like since go list is build-tag constrained, it doesn't include all of the packages/modules that inform go mod tidy and go mod vendor. Is that accurate? Is there a way to use go list to get the full set of dependencies from all build tags?

[bcmills]

go list is indeed tag-constrained. (#42504 tracks the task of addressing that.)