cmd/go: internal error in `go work sync` #65363

trajan0x · 2024-01-30T05:47:41Z

Go version

go version go1.21.5 darwin/arm64

Output of `go env` in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/trajan0x/Library/Caches/go-build'
GOENV='/Users/trajan0x/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/trajan0x/go/pkg/mod'
GONOPROXY='github.com/synapsecns/*'
GONOSUMDB='github.com/synapsecns/*'
GOOS='darwin'
GOPATH='/Users/trajan0x/go'
GOPRIVATE='github.com/synapsecns/*'
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.21.5/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.21.5/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.21.5'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/12/8xtw48x951g0vv4z_ctcr2hr0000gn/T/go-build1424252971=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

git clone https://github.com/synapsecns/sanguine
go work sync
git checkout 995354fca4e89c0931ea5f6f0a77f170aac2f13e (this is happening across a couple commits and I found it on Add native ETH support to relayer synapsecns/sanguine#1840, just providing a single commit on master for reproducibility)

(Note: recipe won't work for this kinda bug, but I created a repl of the repo here and ran the command just for ease of debugging.

What did you see happen?

go: cloud.google.com/go@v0.110.7 requires
        github.com/google/s2a-go@v0.1.4
panic: internal error: found a version conflict, but no constraint it violates

goroutine 1 [running]:
cmd/go/internal/modload.editRequirements({0x1010edbd0, 0x1014830a0}, 0x14013bbaaa0, {0x0, 0x0, 0x1400bb7e828?}, {0x14007bac000, 0x108, 0x1400bb7e800?})
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/internal/modload/edit.go:336 +0x2bc0
cmd/go/internal/modload.EditBuildList({0x1010edbd0, 0x1014830a0}, {0x0, 0x0, 0x0}, {0x14007bac000, 0x108, 0x200})
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/internal/modload/buildlist.go:640 +0x64
cmd/go/internal/workcmd.runSync({0x1010edbd0, 0x1014830a0}, 0x140000265b8?, {0x14?, 0x1010248c0?, 0x100adbc6c?})
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/internal/workcmd/sync.go:117 +0x4d0
main.invoke(0x10143fd80, {0x140000201a0, 0x1, 0x1})
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/main.go:268 +0x4f0
main.main()
        /opt/homebrew/Cellar/go/1.21.5/libexec/src/cmd/go/main.go:186 +0x754

What did you expect to see?

Success, another error, but not a panic.

The fact that this isn't handled gracefully tells me it's probably unintended behavior. But I'm a little unsure based on the context in edit.go

The text was updated successfully, but these errors were encountered:

bcmills · 2024-03-14T20:50:15Z

Some notes:

This does appear to be perfectly reproducible in synapsecns/sanguine@995354f.
github.com/google/s2a-go@v0.1.4 is at go 1.16 and therefore unpruned. It is required directly by several of the modules in the workspace.

When a pruned module has a requirement on another root module, and that root is unpruned, we need to transition from the pruned side of the module graph to the unpruned side. I suspect that isn't happening when it needs to in dqTracker.path.

gopherbot · 2024-03-14T21:01:36Z

Change https://go.dev/cl/571800 mentions this issue: cmd/go/internal/modload: follow dependencies of unpruned roots in dqTracker.path

bcmills · 2024-03-15T16:04:11Z

The bug is triggering in the contrib/terraform-provider-helmproxy module, and the conflict is coming about because the workspace-selected version of github.com/google/pprof is ending up lower than the module-selected version — but github.com/google/pprof isn't relevant to this module anyway.

sanguine/contrib/terraform-provider-helmproxy$ go list -m github.com/google/pprof
github.com/google/pprof v0.0.0-20210506205249-923b5ab0fc1a

sanguine/contrib/terraform-provider-helmproxy$ GOWORK=off go list -m github.com/google/pprof
github.com/google/pprof v0.0.0-20210720184732-4bb14d4b1be1

bcmills · 2024-03-15T17:21:25Z

The dependency on the mismatched pprof comes through an old k8s.io/client-go@v0.25.5:

k8s.io/client-go@v0.25.5 cloud.google.com/go@v0.97.0
cloud.google.com/go@v0.97.0 github.com/googleapis/gax-go/v2@v2.1.0
github.com/googleapis/gax-go/v2@v2.1.0 google.golang.org/api@v0.54.0
google.golang.org/api@v0.54.0 cloud.google.com/go@v0.90.0
cloud.google.com/go@v0.90.0 github.com/google/pprof@v0.0.0-20210720184732-4bb14d4b1be1

But k8s.io/api v0.25.5 is at go 1.19, so it normally has module graph pruning enabled — it must be getting pulled in through some other dependency.

bcmills · 2024-03-15T17:37:27Z

Ok, here's the actual bad path: github.com/hashicorp/terraform-provider-google/v4 v4.2.0 is explicitly required and unpruned (at go 1.16).

From there, we reach the bad pprof unpruned:

github.com/hashicorp/terraform-provider-google/v4@v4.2.0 google.golang.org/api@v0.60.0
google.golang.org/api@v0.60.0 cloud.google.com/go@v0.97.0
cloud.google.com/go@v0.97.0 github.com/googleapis/gax-go/v2@v2.1.0
github.com/googleapis/gax-go/v2@v2.1.0 google.golang.org/api@v0.54.0
google.golang.org/api@v0.54.0 cloud.google.com/go@v0.90.0
cloud.google.com/go@v0.90.0 github.com/google/pprof@v0.0.0-20210720184732-4bb14d4b1be1

So it appears that in workspace mode we are failing to load the transitive dependencies of github.com/hashicorp/terraform-provider-google/v4@v4.2.0.

bcmills · 2024-03-15T17:47:41Z

contrib/tfcore/go.mod is using a wildcard replace directive to replace google.golang.org/api with the source code of a specific version of that module:

	google.golang.org/api => google.golang.org/api v0.86.0

bcmills · 2024-03-15T18:07:53Z

So the skew in the versions seems to come from applying the replace directives inconsistently in go work sync.

When we load the module graph for the first time, we use the replace directives from the workspace overall: https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/workcmd/sync.go;l=57-60;drc=ce8146ed3361f584ba79427ac6c6d6fe9c297bea
But when we later reload each module in the workspace individually, we end up using only the replace directives of the individual module: https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/workcmd/sync.go;l=105-117;drc=ce8146ed3361f584ba79427ac6c6d6fe9c297bea

Moreover, the error returned from EditBuildList causes go work sync to silently continue instead of erroring out:
https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/workcmd/sync.go;l=117-120;drc=ce8146ed3361f584ba79427ac6c6d6fe9c297bea

bcmills · 2024-03-15T18:09:34Z

So, to summarize:

There is one bug in how editRequirements applies module graph pruning when diagnosing unresolvable conflicts, which should be fixed by https://go.dev/cl/571800.
There is another bug in how go work sync applies replace directives, which will need a separate fix.

The combination of the two bugs is what produces the observed symptoms.

…racker.path For #65363. Change-Id: I82ae1098b00c8772ef8d3aa92197e7d8c66d1b37 Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-longtest,gotip-windows-amd64-longtest Reviewed-on: https://go-review.googlesource.com/c/go/+/571800 Reviewed-by: Michael Matloob <matloob@golang.org> Auto-Submit: Michael Matloob <matloob@golang.org> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

matloob · 2024-06-24T20:07:32Z

We weren't able to get this done in time for 1.23, and the original error in the first issue comment isn't showing up anymore so we're going to push this to 1.24.

bcmills changed the title ~~go work sync internal error~~ cmd/go: internal error in go work sync Jan 30, 2024

bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. GoCommand cmd/go modules labels Jan 30, 2024

bcmills self-assigned this Jan 30, 2024

bcmills added this to the Go1.23 milestone Jan 30, 2024

This was referenced Feb 5, 2024

upgrade google terraform provider synapsecns/sanguine#1978

Closed

go work sync fix synapsecns/sanguine#1980

Merged

bcmills added NeedsFix The path to resolution is known, but the work has not been done. FixPending Issues that have a fix which has not yet been reviewed or submitted. labels Mar 14, 2024

gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 14, 2024

bcmills removed their assignment Mar 15, 2024

matloob self-assigned this Apr 29, 2024

matloob modified the milestones: Go1.23, Go1.24 Jun 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: internal error in `go work sync` #65363

cmd/go: internal error in `go work sync` #65363

trajan0x commented Jan 30, 2024 •

edited

Loading

bcmills commented Mar 14, 2024

gopherbot commented Mar 14, 2024

bcmills commented Mar 15, 2024

bcmills commented Mar 15, 2024

bcmills commented Mar 15, 2024

bcmills commented Mar 15, 2024

bcmills commented Mar 15, 2024

bcmills commented Mar 15, 2024 •

edited

Loading

matloob commented Jun 24, 2024

cmd/go: internal error in go work sync #65363

cmd/go: internal error in go work sync #65363

Comments

trajan0x commented Jan 30, 2024 • edited Loading

Go version

Output of go env in your module/workspace:

What did you do?

What did you see happen?

What did you expect to see?

bcmills commented Mar 14, 2024

gopherbot commented Mar 14, 2024

bcmills commented Mar 15, 2024

bcmills commented Mar 15, 2024

bcmills commented Mar 15, 2024

bcmills commented Mar 15, 2024

bcmills commented Mar 15, 2024

bcmills commented Mar 15, 2024 • edited Loading

matloob commented Jun 24, 2024

cmd/go: internal error in `go work sync` #65363

cmd/go: internal error in `go work sync` #65363

trajan0x commented Jan 30, 2024 •

edited

Loading

Output of `go env` in your module/workspace:

bcmills commented Mar 15, 2024 •

edited

Loading