e2e CSV metric is preserved failed intermittently

[akihikokuroda]

Bug Report

e2e CSV metric is preserved failed intermittently
https://github.com/operator-framework/operator-lifecycle-manager/runs/4548853424?check_suite_focus=true#step:4:11624

Metrics are generated for OLM managed resources Given an OperatorGroup that supports all namespaces when a CSV is created when the OLM pod restarts
  CSV metric is preserved
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/metrics_e2e_test.go:151

[BeforeEach] Metrics are generated for OLM managed resources
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/metrics_e2e_test.go:39
[BeforeEach] Given an OperatorGroup that supports all namespaces
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/metrics_e2e_test.go:46
STEP: using the default OperatorGroup created in BeforeSuite
[BeforeEach] when a CSV is created
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/metrics_e2e_test.go:126
 ():
Installing (InstallSucceeded): waiting for install components to report healthy
Succeeded (InstallSucceeded): install strategy completed with no errors
[BeforeEach] when the OLM pod restarts
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/metrics_e2e_test.go:148
[It] CSV metric is preserved
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/metrics_e2e_test.go:151
querying pod operator-lifecycle-manager/olm-operator-64cbf5d7c6-n5b6b
Retrieving metrics using scheme http
[AfterEach] when a CSV is created
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/metrics_e2e_test.go:137


• Failure [4.200 seconds]
Metrics are generated for OLM managed resources
/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/metrics_e2e_test.go:32
  Given an OperatorGroup that supports all namespaces
  /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/metrics_e2e_test.go:45
    when a CSV is created
    /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/metrics_e2e_test.go:121
      when the OLM pod restarts
      /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/metrics_e2e_test.go:147
	CSV metric is preserved [It]
	/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/metrics_e2e_test.go:151

	Expected
            <[]e2e.Metric | len:2, cap:2>: [
                {Family: "csv_count", Labels: nil, Value: 0},
                {
                    Family: "csv_upgrade_count",
                    Labels: nil,
                    Value: 0,
                },
            ]
        to contain element matching
            <*e2e.LikeMetricMatcher | 0xc001a752d8>: {
                Predicates: [
                    {
                        f: 0x2c17960,
                        name: "WithFamily(csv_succeeded)",
                    },
                    {
                        f: 0x2c179e0,
                        name: "WithLabel(name=csv-test-9lf92-stable)",
                    },
                    {f: 0x2c17c20, name: "WithValue(1)"},
                ],
            }

        /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/metrics_e2e_test.go:152

        Full Stack Trace
        github.com/operator-framework/operator-lifecycle-manager/test/e2e.glob..func12.2.3.4.2()
                /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/metrics_e2e_test.go:152 +0x311
        github.com/operator-framework/operator-lifecycle-manager/test/e2e.TestEndToEnd(0xc000653080)
                /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/e2e_test.go:59 +0x2d6
        testing.tRunner(0xc000653080, 0x3470840)
                /opt/hostedtoolcache/go/1.16.10/x64/src/testing/testing.go:1193 +0xef
        created by testing.(*T).Run
                /opt/hostedtoolcache/go/1.16.10/x64/src/testing/testing.go:1238 +0x2b3

What did you do?
This error can be reproduced more often locally by adding a short delay in cmd/olm/main.go

diff --git a/cmd/olm/main.go b/cmd/olm/main.go
index efb1e00e0..08892a165 100644
--- a/cmd/olm/main.go
+++ b/cmd/olm/main.go
@@ -170,7 +170,7 @@ func main() {

        op.Run(ctx)
        <-op.Ready()
-
+time.Sleep(5*time.Second)
        // Emit CSV metric
        if err = op.EnsureCSVMetric(); err != nil {
		logger.WithError(err).Fatal("error emitting metrics for existing CSV")

What did you expect to see?
No error

What did you see instead? Under which circumstances?
Error above

Environment

• operator-lifecycle-manager version: master branch

• Kubernetes version information:

• Kubernetes cluster kind:

Possible Solution

Additional context
Add any other context about the problem here.

[timflannagan]

Closing this as a dup of #2529 but I'll update your PR to update the issue that was linked. Thanks for opening this (and the corresponding PR) - it's super appreciated.