proxy.golang.org: not found: fetch timed out

[bitoku]

$ go version
go version go1.21.5 linux/amd64

`go env`

go env
GO111MODULE='on'
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/azureuser/.cache/go-build'
GOENV='/home/azureuser/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/azureuser/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/azureuser/go'
GOPRIVATE=''
GOPROXY='direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.5'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='0'
GOMOD='/home/azureuser/ARO-Installer/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build3551886818=/tmp/go-build -gno-record-gcc-switches'

I got the error when I run go get.

$ go get github.com/openshift/installer-aro@release-4.14-azure
go: downloading github.com/openshift/installer-aro v0.9.0-master.0.20240214010831-fd48bf30c1af
go: github.com/openshift/installer-aro@release-4.14-azure: github.com/openshift/installer-aro@v0.9.0-master.0.20240214010831-fd48bf30c1af: verifying module: github.com/openshift/installer-aro@v0.9.0-master.0.20240214010831-fd48bf30c1af: reading https://sum.golang.org/lookup/github.com/openshift/installer-aro@v0.9.0-master.0.20240214010831-fd48bf30c1af: 404 Not Found
        server response: not found: fetch timed out

Several hours have passed since we create that version but it isn't available yet.

#57185 #56174 might be related but not sure.
The question is, is this just a temporary issue or is there something we should do?

[seankhliao]

maybe the module is too big?
I see it falls back to direct to download the module, but that won't work with sumdb because it still relies on proxy.golang.org

$  go get -v -x github.com/openshift/installer-aro@release-4.14-azure
# get https://proxy.golang.org/github.com/@v/release-4.14-azure.info
# get https://proxy.golang.org/github.com/openshift/installer-aro/@v/release-4.14-azure.info
# get https://proxy.golang.org/github.com/openshift/@v/release-4.14-azure.info
# get https://proxy.golang.org/github.com/openshift/installer-aro/@v/release-4.14-azure.info: 404 Not Found (0.128s)
# get https://proxy.golang.org/github.com/openshift/@v/release-4.14-azure.info: 404 Not Found (0.135s)
# get https://proxy.golang.org/github.com/@v/release-4.14-azure.info: 404 Not Found (0.167s)
mkdir -p /home/user/.data/go/pkg/mod/cache/vcs # git3 https://github.com/openshift/installer-aro
# lock /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a.lock
# /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a for git3 https://github.com/openshift/installer-aro
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git tag -l
0.005s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git tag -l
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git ls-remote -q origin
1.624s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git ls-remote -q origin
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' fd48bf30c1af8469c8774eabe41c58e0b33d740f --
0.004s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' fd48bf30c1af8469c8774eabe41c58e0b33d740f --
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git fetch -f -c protocol.version=2 --depth=1 origin refs/heads/release-4.14-azure:refs/heads/release-4.14-azure
0.003s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git fetch -f -c protocol.version=2 --depth=1 origin refs/heads/release-4.14-azure:refs/heads/release-4.14-azure
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git fetch -f origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
x93.110s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git fetch -f origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' fd48bf30c1af8469c8774eabe41c58e0b33d740f --
0.003s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' fd48bf30c1af8469c8774eabe41c58e0b33d740f --
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' e3fceacc975953f56cb09931e6be015a36eb6075 --
0.003s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' e3fceacc975953f56cb09931e6be015a36eb6075 --
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git cat-file blob e3fceacc975953f56cb09931e6be015a36eb6075:go.mod
0.001s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git cat-file blob e3fceacc975953f56cb09931e6be015a36eb6075:go.mod
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git cat-file blob e3fceacc975953f56cb09931e6be015a36eb6075:go.mod
0.001s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git cat-file blob e3fceacc975953f56cb09931e6be015a36eb6075:go.mod
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' fd48bf30c1af8469c8774eabe41c58e0b33d740f --
0.003s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' fd48bf30c1af8469c8774eabe41c58e0b33d740f --
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git for-each-ref --format %(refname) refs/tags --merged fd48bf30c1af8469c8774eabe41c58e0b33d740f
0.081s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git for-each-ref --format %(refname) refs/tags --merged fd48bf30c1af8469c8774eabe41c58e0b33d740f
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' fd48bf30c1af --
0.003s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' fd48bf30c1af --
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git cat-file blob fd48bf30c1af8469c8774eabe41c58e0b33d740f:go.mod
0.002s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git cat-file blob fd48bf30c1af8469c8774eabe41c58e0b33d740f:go.mod
# get https://proxy.golang.org/github.com/openshift/installer-aro/@v/list
# get https://proxy.golang.org/github.com/openshift/installer-aro/@v/list: 200 OK (0.113s)
# get https://proxy.golang.org/github.com/openshift/installer-aro/@v/v0.16.1.info
# get https://proxy.golang.org/github.com/openshift/installer-aro/@v/v0.16.1.info: 200 OK (0.032s)
# get https://proxy.golang.org/github.com/openshift/installer-aro/@v/v0.16.1.mod
# get https://proxy.golang.org/github.com/openshift/installer-aro/@v/v0.16.1.mod: 200 OK (0.071s)
# get https://proxy.golang.org/sumdb/sum.golang.org/supported
# get https://proxy.golang.org/sumdb/sum.golang.org/supported: 404 Not Found (0.024s)
# get https://sum.golang.org/lookup/github.com/openshift/installer-aro@v0.16.1
# get https://sum.golang.org/lookup/github.com/openshift/installer-aro@v0.16.1: 200 OK (0.185s)
# get https://sum.golang.org/tile/8/0/x088/834
# get https://sum.golang.org/tile/8/1/347.p/197
# get https://sum.golang.org/tile/8/0/x089/029.p/170
# get https://sum.golang.org/tile/8/0/x088/834: 200 OK (0.031s)
# get https://sum.golang.org/tile/8/1/347.p/197: 200 OK (0.037s)
# get https://sum.golang.org/tile/8/0/x089/029.p/170: 200 OK (0.078s)
# get https://sum.golang.org/tile/8/0/x085/524
# get https://sum.golang.org/tile/8/0/x085/524: 200 OK (0.045s)
go: downloading github.com/openshift/installer-aro v0.9.0-master.0.20240214010831-fd48bf30c1af
# get https://proxy.golang.org/github.com/openshift/installer-aro/@v/v0.9.0-master.0.20240214010831-fd48bf30c1af.zip
# get https://proxy.golang.org/github.com/openshift/installer-aro/@v/v0.9.0-master.0.20240214010831-fd48bf30c1af.zip: 404 Not Found (58.043s)
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git cat-file blob fd48bf30c1af8469c8774eabe41c58e0b33d740f:go.mod
0.004s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git cat-file blob fd48bf30c1af8469c8774eabe41c58e0b33d740f:go.mod
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git merge-base --is-ancestor -- v0.9.0-master fd48bf30c1af8469c8774eabe41c58e0b33d740f
0.059s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git merge-base --is-ancestor -- v0.9.0-master fd48bf30c1af8469c8774eabe41c58e0b33d740f
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git cat-file blob fd48bf30c1af8469c8774eabe41c58e0b33d740f:go.mod
0.002s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git cat-file blob fd48bf30c1af8469c8774eabe41c58e0b33d740f:go.mod
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git cat-file blob fd48bf30c1af8469c8774eabe41c58e0b33d740f:go.mod
0.002s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git cat-file blob fd48bf30c1af8469c8774eabe41c58e0b33d740f:go.mod
cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ fd48bf30c1af8469c8774eabe41c58e0b33d740f
^[[122;9u21.776s # cd /home/user/.data/go/pkg/mod/cache/vcs/c19bff20f376595d3737e49de02a1a8348d373f213b975e6a811d3818b33e08a; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ fd48bf30c1af8469c8774eabe41c58e0b33d740f
# get https://sum.golang.org/lookup/github.com/openshift/installer-aro@v0.9.0-master.0.20240214010831-fd48bf30c1af
# get https://sum.golang.org/lookup/github.com/openshift/installer-aro@v0.9.0-master.0.20240214010831-fd48bf30c1af: 404 Not Found (52.426s)
go: github.com/openshift/installer-aro@release-4.14-azure: github.com/openshift/installer-aro@v0.9.0-master.0.20240214010831-fd48bf30c1af: verifying module: github.com/openshift/installer-aro@v0.9.0-master.0.20240214010831-fd48bf30c1af: reading https://sum.golang.org/lookup/github.com/openshift/installer-aro@v0.9.0-master.0.20240214010831-fd48bf30c1af: 404 Not Found
	server response: not found: 

[thanm]

This is reproducible on my end as well.

cc @golang/tools-team

[patrickdillon]

I am a maintainer for openshift/installer, of which the OP is a fork, and we're hitting the same issue (and then some).

The problem seems to be the size of the commit history. Prior to my debugging over the last two days, our most recent tag was over 5 years old and was more than 10,000 commits behind master. I was able to mitigate the issue by retroactively tagging some of our commit history.

I have captured my debugging in OCPBUGS-42448, which should be publicly viewable.

It would be helpful to have more insight into this issue.