Bug 2081483: Emit csv_succeeded metrics on startup #2768

anik120 · 2022-05-03T20:13:26Z

Description of the change:

Backport of openshift/operator-framework-olm#288

Motivation for the change:

Reviewer Checklist

Implementation matches the proposed design, or proposal is updated to match implementation
Sufficient unit test coverage
Sufficient end-to-end test coverage
Docs updated or added to /doc
Commit messages sensible and descriptive
Tests marked as [FLAKE] are truly flaky
Tests that remove the [FLAKE] tag are no longer flaky

Problem: The getMetricsFromPod function assumes that metrics are exposed on port 8080. This function fails to retrieve metrics from the olm or catalog operator when the port is changed. Solution: Name the port in each of the deployments and update the getMetricsFromPod function to infer the port number from the deployments. Signed-off-by: Alexander Greene <greene.al1991@gmail.com>

openshift-ci · 2022-05-03T20:13:28Z

@anik120: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

Emit csv_succeeded metrics on startup

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Signed-off-by: Josef Karasek <jkarasek@redhat.com>

Signed-off-by: akihikokuroda <akihikokuroda2020@gmail.com>

anik120 · 2022-05-03T20:39:49Z

/test unit

openshift-ci · 2022-05-03T21:13:48Z

@anik120: This pull request references Bugzilla bug 2081483, which is invalid:

expected dependent Bugzilla bug 2072995 to target a release in 4.8.0, 4.8.z, but it targets "4.9.z" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 2081483: Emit csv_succeeded metrics on startup

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

anik120 · 2022-05-03T21:15:41Z

/bugzilla refresh

openshift-ci · 2022-05-03T21:15:51Z

@anik120: This pull request references Bugzilla bug 2081483, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.z) matches configured target release for branch (4.7.z)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 2074680 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
dependent Bugzilla bug 2074680 targets the "4.8.z" release, which is one of the valid target releases: 4.8.0, 4.8.z
bug has dependents

Requesting review from QA contact:
/cc @jianzhangbjz

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-05-03T21:15:53Z

@openshift-ci[bot]: GitHub didn't allow me to request PR reviews from the following users: jianzhangbjz.

Note that only operator-framework members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

@anik120: This pull request references Bugzilla bug 2081483, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug

bug is open, matching expected state (open)

bug target release (4.7.z) matches configured target release for branch (4.7.z)

bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

dependent bug Bugzilla bug 2074680 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))

dependent Bugzilla bug 2074680 targets the "4.8.z" release, which is one of the valid target releases: 4.8.0, 4.8.z

bug has dependents

Requesting review from QA contact:
/cc @jianzhangbjz

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

bandrade · 2022-05-05T04:52:10Z

/label cherry-pick-approved
/lgtm

oc get clusterversion                        
NAME      VERSION                                                  AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.ci.test-2022-05-05-033547-ci-ln-91hsckb-latest   True        False         16m     Cluster version is 4.7.0-0.ci.test-2022-05-05-033547-ci-ln-91hsckb-latest


oc exec catalog-operator-5954984694-fndf7  -n openshift-operator-lifecycle-manager -- olm --version
OLM version: 0.17.0
git commit: 706fdafce9dc4af8273b54188145ee978f1751fd

1, install some operators
oc get csv -A | grep -v elasticsearch | grep -v nginx-ingress | grep -v must-gather                
NAMESPACE                              NAME                        DISPLAY                     VERSION    REPLACES   PHASE
openshift-logging                      cluster-logging.5.2.9-42    Red Hat OpenShift Logging   5.2.9-42              Succeeded
openshift-operator-lifecycle-manager   packageserver               Package Server              0.17.0                Succeeded
test-1                                 anzo-operator               Anzo Operator               2.0.1                 Succeeded
test-3                                 cockroach-operator.v2.6.0   CockroachDB Operator        2.6.0                 Succeed


2, port-fowarding to the olm-operator pod and curling the metrics endpoint
oc port-forward olm-operator-58cd69f5f6-gxzb4  8081 -n openshift-operator-lifecycle-manager
Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081
Handling connection for 8081

curl -k https://localhost:8081/metrics | grep csv  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6890    0  6890    0     0  11579      0 --# HELP csv_count Number of CSVs successfully registered
:--:-# TYPE csv_count gauge
- csv_count 4
--:-# HELP csv_succeeded Successful CSV install
-:# TYPE csv_succeeded gauge
-- csv_succeeded{name="anzo-operator",namespace="test-1",version="2.0.1"} 1
--:csv_succeeded{name="cluster-logging.5.2.9-42",namespace="openshift-logging",version="5.2.9-42"} 1
--csv_succeeded{name="cockroach-operator.v2.6.0",namespace="test-3",version="2.6.0"} 1
:--csv_succeeded{name="packageserver",namespace="openshift-operator-lifecycle-manager",version="0.17.0"} 1
 1# HELP csv_upgrade_count Monotonic count of CSV upgrades
15# TYPE csv_upgrade_count counter
6csv_upgrade_count 0


oc delete pod olm-operator-58cd69f5f6-gxzb4 -n openshift-operator-lifecycle-manager        
pod "olm-operator-58cd69f5f6-gxzb4" deleted

oc port-forward olm-operator-58cd69f5f6-9prm7  8081 -n openshift-operator-lifecycle-manager                                                         
Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081

url -k https://localhost:8081/metrics | grep csv  
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP csv_count Number of CSVs successfully registered
# TYPE csv_count gauge
csv_count 4
# HELP csv_succeeded Successful CSV install
# TYPE csv_succeeded gauge
csv_succeeded{name="anzo-operator",namespace="test-1",version="2.0.1"} 1
csv_succeeded{name="cluster-logging.5.2.9-42",namespace="openshift-logging",version="5.2.9-42"} 1
csv_succeeded{name="cockroach-operator.v2.6.0",namespace="test-3",version="2.6.0"} 1
csv_succeeded{name="packageserver",namespace="openshift-operator-lifecycle-manager",version="0.17.0"} 1
# HELP csv_upgrade_count Monotonic count of CSV upgrades
# TYPE csv_upgrade_count counter
csv_upgrade_count 0
100  6891    0  6891    0     0  11601      0 --:--:-- --:--:-- --:--:-- 11601

openshift-ci · 2022-05-05T04:52:12Z

@bandrade: changing LGTM is restricted to collaborators

In response to this:

/label cherry-pick-approved
/lgtm

oc get clusterversion                        
NAME      VERSION                                                  AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.ci.test-2022-05-05-033547-ci-ln-91hsckb-latest   True        False         16m     Cluster version is 4.7.0-0.ci.test-2022-05-05-033547-ci-ln-91hsckb-latest


oc exec catalog-operator-5954984694-fndf7  -n openshift-operator-lifecycle-manager -- olm --version
OLM version: 0.17.0
git commit: 706fdafce9dc4af8273b54188145ee978f1751fd

1, install some operators
oc get csv -A | grep -v elasticsearch | grep -v nginx-ingress | grep -v must-gather                
NAMESPACE                              NAME                        DISPLAY                     VERSION    REPLACES   PHASE
openshift-logging                      cluster-logging.5.2.9-42    Red Hat OpenShift Logging   5.2.9-42              Succeeded
openshift-operator-lifecycle-manager   packageserver               Package Server              0.17.0                Succeeded
test-1                                 anzo-operator               Anzo Operator               2.0.1                 Succeeded
test-3                                 cockroach-operator.v2.6.0   CockroachDB Operator        2.6.0                 Succeed


2, port-fowarding to the olm-operator pod and curling the metrics endpoint
oc port-forward olm-operator-58cd69f5f6-gxzb4  8081 -n openshift-operator-lifecycle-manager
Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081
Handling connection for 8081

curl -k https://localhost:8081/metrics | grep csv  
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100  6890    0  6890    0     0  11579      0 --# HELP csv_count Number of CSVs successfully registered
:--:-# TYPE csv_count gauge
- csv_count 4
--:-# HELP csv_succeeded Successful CSV install
-:# TYPE csv_succeeded gauge
-- csv_succeeded{name="anzo-operator",namespace="test-1",version="2.0.1"} 1
--:csv_succeeded{name="cluster-logging.5.2.9-42",namespace="openshift-logging",version="5.2.9-42"} 1
--csv_succeeded{name="cockroach-operator.v2.6.0",namespace="test-3",version="2.6.0"} 1
:--csv_succeeded{name="packageserver",namespace="openshift-operator-lifecycle-manager",version="0.17.0"} 1
1# HELP csv_upgrade_count Monotonic count of CSV upgrades
15# TYPE csv_upgrade_count counter
6csv_upgrade_count 0


oc delete pod olm-operator-58cd69f5f6-gxzb4 -n openshift-operator-lifecycle-manager        
pod "olm-operator-58cd69f5f6-gxzb4" deleted

oc port-forward olm-operator-58cd69f5f6-9prm7  8081 -n openshift-operator-lifecycle-manager                                                         
Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081

url -k https://localhost:8081/metrics | grep csv  
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
 0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP csv_count Number of CSVs successfully registered
# TYPE csv_count gauge
csv_count 4
# HELP csv_succeeded Successful CSV install
# TYPE csv_succeeded gauge
csv_succeeded{name="anzo-operator",namespace="test-1",version="2.0.1"} 1
csv_succeeded{name="cluster-logging.5.2.9-42",namespace="openshift-logging",version="5.2.9-42"} 1
csv_succeeded{name="cockroach-operator.v2.6.0",namespace="test-3",version="2.6.0"} 1
csv_succeeded{name="packageserver",namespace="openshift-operator-lifecycle-manager",version="0.17.0"} 1
# HELP csv_upgrade_count Monotonic count of CSV upgrades
# TYPE csv_upgrade_count counter
csv_upgrade_count 0
100  6891    0  6891    0     0  11601      0 --:--:-- --:--:-- --:--:-- 11601

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2022-05-05T04:52:18Z

@bandrade: Can not set label cherry-pick-approved: Must be member in one of these teams: [openshift-patch-managers]

In response to this:

/label cherry-pick-approved
/lgtm

oc get clusterversion                        
NAME      VERSION                                                  AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.7.0-0.ci.test-2022-05-05-033547-ci-ln-91hsckb-latest   True        False         16m     Cluster version is 4.7.0-0.ci.test-2022-05-05-033547-ci-ln-91hsckb-latest


oc exec catalog-operator-5954984694-fndf7  -n openshift-operator-lifecycle-manager -- olm --version
OLM version: 0.17.0
git commit: 706fdafce9dc4af8273b54188145ee978f1751fd

1, install some operators
oc get csv -A | grep -v elasticsearch | grep -v nginx-ingress | grep -v must-gather                
NAMESPACE                              NAME                        DISPLAY                     VERSION    REPLACES   PHASE
openshift-logging                      cluster-logging.5.2.9-42    Red Hat OpenShift Logging   5.2.9-42              Succeeded
openshift-operator-lifecycle-manager   packageserver               Package Server              0.17.0                Succeeded
test-1                                 anzo-operator               Anzo Operator               2.0.1                 Succeeded
test-3                                 cockroach-operator.v2.6.0   CockroachDB Operator        2.6.0                 Succeed


2, port-fowarding to the olm-operator pod and curling the metrics endpoint
oc port-forward olm-operator-58cd69f5f6-gxzb4  8081 -n openshift-operator-lifecycle-manager
Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081
Handling connection for 8081

curl -k https://localhost:8081/metrics | grep csv  
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
100  6890    0  6890    0     0  11579      0 --# HELP csv_count Number of CSVs successfully registered
:--:-# TYPE csv_count gauge
- csv_count 4
--:-# HELP csv_succeeded Successful CSV install
-:# TYPE csv_succeeded gauge
-- csv_succeeded{name="anzo-operator",namespace="test-1",version="2.0.1"} 1
--:csv_succeeded{name="cluster-logging.5.2.9-42",namespace="openshift-logging",version="5.2.9-42"} 1
--csv_succeeded{name="cockroach-operator.v2.6.0",namespace="test-3",version="2.6.0"} 1
:--csv_succeeded{name="packageserver",namespace="openshift-operator-lifecycle-manager",version="0.17.0"} 1
1# HELP csv_upgrade_count Monotonic count of CSV upgrades
15# TYPE csv_upgrade_count counter
6csv_upgrade_count 0


oc delete pod olm-operator-58cd69f5f6-gxzb4 -n openshift-operator-lifecycle-manager        
pod "olm-operator-58cd69f5f6-gxzb4" deleted

oc port-forward olm-operator-58cd69f5f6-9prm7  8081 -n openshift-operator-lifecycle-manager                                                         
Forwarding from 127.0.0.1:8081 -> 8081
Forwarding from [::1]:8081 -> 8081

url -k https://localhost:8081/metrics | grep csv  
 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
 0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0# HELP csv_count Number of CSVs successfully registered
# TYPE csv_count gauge
csv_count 4
# HELP csv_succeeded Successful CSV install
# TYPE csv_succeeded gauge
csv_succeeded{name="anzo-operator",namespace="test-1",version="2.0.1"} 1
csv_succeeded{name="cluster-logging.5.2.9-42",namespace="openshift-logging",version="5.2.9-42"} 1
csv_succeeded{name="cockroach-operator.v2.6.0",namespace="test-3",version="2.6.0"} 1
csv_succeeded{name="packageserver",namespace="openshift-operator-lifecycle-manager",version="0.17.0"} 1
# HELP csv_upgrade_count Monotonic count of CSV upgrades
# TYPE csv_upgrade_count counter
csv_upgrade_count 0
100  6891    0  6891    0     0  11601      0 --:--:-- --:--:-- --:--:-- 11601

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jianzhangbjz · 2022-05-05T13:27:09Z

/label cherry-pick-approved

openshift-ci · 2022-05-05T13:27:18Z

@jianzhangbjz: Can not set label cherry-pick-approved: Must be member in one of these teams: [openshift-patch-managers]

In response to this:

/label cherry-pick-approved

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

timflannagan · 2022-05-06T13:40:30Z

/bugzilla-refresh

timflannagan · 2022-05-06T13:40:49Z

Manually adding the CPA label after QE ack in #2768 (comment).

timflannagan · 2022-05-06T13:41:15Z

/bugzilla refresh

openshift-ci · 2022-05-06T13:41:21Z

@timflannagan: This pull request references Bugzilla bug 2081483, which is valid.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.7.z) matches configured target release for branch (4.7.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 2074680 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
dependent Bugzilla bug 2074680 targets the "4.8.z" release, which is one of the valid target releases: 4.8.0, 4.8.z
bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (bandrade@redhat.com), skipping review request.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

timflannagan · 2022-05-09T19:13:35Z

/lgtm

dinhxuanvu

/approve

kevinrizza · 2022-05-09T23:18:05Z

/approve

openshift-ci · 2022-05-09T23:18:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anik120, dinhxuanvu, kevinrizza

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [kevinrizza]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

perdasilva · 2022-05-11T19:51:34Z

/backport-risk-assessed

timflannagan · 2022-05-12T01:13:49Z

Manually adding the backport-risk-assessed label after Per's ACK in #2768 (comment). It looks like that command needs to be prefixed with /label ... 😅

openshift-ci · 2022-05-12T01:15:54Z

@anik120: All pull requests linked via external trackers have merged:

operator-framework/operator-lifecycle-manager#2768

Bugzilla bug 2081483 has been moved to the MODIFIED state.

In response to this:

Bug 2081483: Emit csv_succeeded metrics on startup

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot requested review from exdx and njhale May 3, 2022 20:13

anik120 mentioned this pull request May 3, 2022

4.7 csv metrics openshift/operator-framework-olm#301

Closed

Josef Karasek and others added 2 commits May 3, 2022 16:17

Emit CSV metric on startup (operator-framework#2216)

239c027

Signed-off-by: Josef Karasek <jkarasek@redhat.com>

fix e2e CSV metric is preserved failure (operator-framework#2530)

5cf1d5b

Signed-off-by: akihikokuroda <akihikokuroda2020@gmail.com>

anik120 force-pushed the 4.7-csv-succeeded-fix branch from 1d3acd1 to 5cf1d5b Compare May 3, 2022 20:17

anik120 changed the title ~~Emit csv_succeeded metrics on startup~~ Bug 2081483: Emit csv_succeeded metrics on startup May 3, 2022

openshift-ci bot added the bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. label May 3, 2022

openshift-ci bot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label May 3, 2022

openshift-ci bot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label May 3, 2022

openshift-ci bot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label May 3, 2022

timflannagan added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label May 6, 2022

openshift-ci bot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. and removed bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. labels May 6, 2022

openshift-ci bot assigned timflannagan May 9, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 9, 2022

dinhxuanvu approved these changes May 9, 2022

View reviewed changes

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 9, 2022

timflannagan added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label May 12, 2022

openshift-merge-robot merged commit 64cff34 into operator-framework:release-4.7 May 12, 2022

Bug 2081483: Emit csv_succeeded metrics on startup #2768

Bug 2081483: Emit csv_succeeded metrics on startup #2768

Uh oh!

Conversation

anik120 commented May 3, 2022

Uh oh!

openshift-ci bot commented May 3, 2022

Uh oh!

anik120 commented May 3, 2022

Uh oh!

openshift-ci bot commented May 3, 2022

Uh oh!

anik120 commented May 3, 2022

Uh oh!

openshift-ci bot commented May 3, 2022

Uh oh!

openshift-ci bot commented May 3, 2022

Uh oh!

bandrade commented May 5, 2022

Uh oh!

openshift-ci bot commented May 5, 2022

Uh oh!

openshift-ci bot commented May 5, 2022

Uh oh!

jianzhangbjz commented May 5, 2022

Uh oh!

openshift-ci bot commented May 5, 2022

Uh oh!

timflannagan commented May 6, 2022

Uh oh!

timflannagan commented May 6, 2022

Uh oh!

timflannagan commented May 6, 2022

Uh oh!

openshift-ci bot commented May 6, 2022

Uh oh!

timflannagan commented May 9, 2022

Uh oh!

dinhxuanvu left a comment

Choose a reason for hiding this comment

Uh oh!

kevinrizza commented May 9, 2022

Uh oh!

openshift-ci bot commented May 9, 2022

Uh oh!

perdasilva commented May 11, 2022

Uh oh!

timflannagan commented May 12, 2022

Uh oh!

openshift-ci bot commented May 12, 2022

Uh oh!

Uh oh!