[admin] Usage of GOPROXY in releases

[mx-psi]

During the v0.76.x we faced some issues with the Go proxy that delayed the release and caused confusion. We could disable the Go proxy by using GOPROXY=direct, although this would come at the expense of speed and possibly reliability. Another disadvantage is that we won't put the released libraries up in Go proxy ourselves, but rather wait until the first usage of them.

Even if we don't disable the Go proxy on release jobs, we may want to document that disabling the Go proxy may be useful during the release in the Troubleshooting section. (Done in #7597)

Relates to #7576

[mx-psi]

cc @open-telemetry/collector-approvers @open-telemetry/collector-contrib-approvers I will bring this up on the next Collector SIG if we haven't decided then, but would appreciate if you have an opinion about this.

[TylerHelmuth]

@mx-psi I have PR open for another pass through the release doc that does add GOPROXY=direct to the troubleshooting section: #7597

[mx-psi]

@TylerHelmuth awesome, let's keep this open only for whether we should add GOPROXY=direct in CI or not then.

[Aneurysm9]

I would definitely support having a CI workflow using GOPROXY=direct. I am required to use that setting and frequently run into issues with dependencies that have modified tags that are masked by the use of the proxy. It always seems to happen at the worst possible time, so any mechanism that could drive earlier detection of such issues would be appreciated.

[mx-psi]

One concern brought up by @MovieStoreGuy is that we might get rate limited by Github. Authenticating on the workflow (see here) may help here (although I don't see anything explicit on the docs?).

Per @bogdandrutu, we should ask on golang/go first for guidance. I will sync with @djaglowski about the specific problems we had.

[seankhliao]

From the related issues filed at the same time, I guess this is due to proxy.golang.org's 30min negative cache time? Upstream are unlikely to change it for now, ref: golang/go#42809

If the only issue was with collector components, you could consider the below config,
which would allow using the proxy for most dependencies, but use git directly for the collector components

GOPROXY='https://proxy.golang.org,direct' # default
GOPRIVATE='go.opentelemetry.io/collector,github.com/open-telemetry/opentelemetry-collector-contrib'

[mx-psi]

Thanks @seankhliao, I think we should try that. On the v0.76.x release cycle we were getting a 404 (concretely):

go: downloading go.opentelemetry.io/collector/pdata v1.0.0-rcv0011
go: go.opentelemetry.io/collector/pdata@v1.0.0-rcv0011: verifying module: go.opentelemetry.io/collector/pdata@v1.0.0-rcv0011: reading https://sum.golang.org/lookup/go.opentelemetry.io/collector/pdata@v1.0.0-rcv0011: 404 Not Found
        server response: not found: go.opentelemetry.io/collector/pdata@v1.0.0-rcv0011: invalid version: unknown revision pdata/v1.0.0-rcv0011

but it does sound like the same kind of issue. One of the Go team members on that issue now works at my company so I will reach out to them in private as well about this to confirm :)

[mx-psi]

I followed up with @katiehockman who used to work on this. An alternative to avoiding the Go proxy is to ensure we wait a few minutes to avoid the negative cache (the proxy caches the 404 for ~30 minutes max).

To sum up the discussion, we have four options so far:

0. Option 0: Do nothing, keep the current state.
1. Option 1: Do not use the Go proxy altogether in releases
2. Option 2: Do not use the Go proxy, but only for OpenTelemetry Collector modules (see [admin] Usage of GOPROXY in releases #7629 (comment))
3. Option 3: Introduce waiting time to avoid negative caching

And there are four dimensions that we can judge them by:

• Whether we test for GOPROXY=direct users as mentioned in [admin] Usage of GOPROXY in releases #7629 (comment)
• Whether we hit the negative cache issues that we are (pressumably) currently facing
• The expected release speed, when compared with the current baseline
• The expected release reliability, when compared with the current baseline

This table summarizes the dimension's values for each option and the current state:

	Test of GOPROXY=direct?	Negative cache issues?	Release speed	Release reliability
Current state	No	Yes	Baseline	Baseline
Option 1 (no proxy)	Yes	No	Slowest	Least reliable
Option 2 (some proxy)	No	No	~Baseline	~Baseline
Option 3 (just wait!)	No	No	Slower	~Baseline

[mx-psi]

This happened again during the v0.111.0 release