Skip to content

OCPBUGS-55450: Only populate Status.PinnedImageSets.CurrentGeneration on valid generation value #5023

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 8, 2025

Conversation

isabella-janssen
Copy link
Member

@isabella-janssen isabella-janssen commented May 1, 2025

Closes: OCPBUGS-55450

- What I did
This adds a check to only set Status.PinnedImageSets.CurrentGeneration in a MachineConfigNode object when the generation value is valid, or non-zero.

- How to verify it

  1. Launch a cluster with this PR.
  2. Apply an invalid PIS.
Example Invalid PIS targeting "custom" MCP
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
  name: test-pinned
  labels:
    machineconfiguration.openshift.io/role: "custom"
spec:
  pinnedImages:
   - name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86d26e7ebcccd6f07a75db5b1e56283b25c2ee1c6a755d6ffc5a4d59beb9cdef
3. Check that the PIS reference in the MCN has values for `Name`, `DesiredGeneration`, `LastFailedGeneration`, and `LastFailedGenerationError`.
...
Status:
  Conditions:
    Last Transition Time:  2025-05-07T13:11:46Z
    Message:               node is prefetching images: ip-10-0-46-80.ec2.internal
    Reason:                ImagePrefetch
    Status:                True
    Type:                  PinnedImageSetsProgressing
...
    Last Transition Time:  2025-05-07T13:11:48Z
    Message:               One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details
    Reason:                PrefetchFailed
    Status:                True
    Type:                  PinnedImageSetsDegraded
...
Pinned Image Sets:
    Desired Generation:            1
    Last Failed Generation:        1
    Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86d26e7ebcccd6f07a75db5b1e56283b25c2ee1c6a755d6ffc5a4d59beb9cdef": exit status 125
    Name:                          test-pinned
  1. Apply a valid PIS.
Example valid PIS targeting "custom" MCP
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
  name: test-pinned
  labels:
    machineconfiguration.openshift.io/role: "custom"
spec:
  pinnedImages:
   - name: quay.io/openshifttest/busybox@sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1
  1. Check that the PIS reference in the MCN has values for Name, DesiredGeneration, and CurrentGeneration.
...
Status:
  Conditions:
    Last Transition Time:  2025-05-07T13:21:19Z
    Message:               All pinned image sets complete
    Reason:                AsExpected
    Status:                False
    Type:                  PinnedImageSetsProgressing
...
Last Transition Time:  2025-05-07T13:21:19Z
    Message:               All is good
    Reason:                AsExpected
    Status:                False
    Type:                  PinnedImageSetsDegraded
...
  Pinned Image Sets:
    Current Generation:  2
    Desired Generation:  2
    Name:                test-pinned
  1. Check that there are no error logs (especially those starting with "Error applying MCN status").

Note: To see the original failures, see this example payload job: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-machine-config-operator-5034-ci-4.19-e2e-gcp-ovn-techpreview-serial/1919889377152995328.

- Description for the changelog
OCPBUGS-55450: Only populate Status.PinnedImageSets.CurrentGeneration on valid generation value

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 1, 2025
Copy link
Contributor

openshift-ci bot commented May 1, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 1, 2025
@isabella-janssen isabella-janssen changed the title (WIP) Debugging PIS/MCN OCPBUGS-55450: Only populate Status.PinnedImageSets.CurrentGeneration on valid generation value May 6, 2025
@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. labels May 6, 2025
@openshift-ci-robot
Copy link
Contributor

@isabella-janssen: This pull request references Jira Issue OCPBUGS-55450, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.0) matches configured target version for branch (4.20.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

- What I did

- How to verify it

- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label May 6, 2025
@openshift-ci openshift-ci bot requested a review from sergiordlr May 6, 2025 19:36
@openshift-ci-robot
Copy link
Contributor

@isabella-janssen: This pull request references Jira Issue OCPBUGS-55450, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.20.0) matches configured target version for branch (4.20.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

In response to this:

Closes: OCPBUGS-55450

- What I did
This adds a check to only set Status.PinnedImageSets.CurrentGeneration in a MachineConfigNode object when the generation value is valid, or non-zero.

- How to verify it

- Description for the changelog
OCPBUGS-55450: Only populate Status.PinnedImageSets.CurrentGeneration on valid generation value

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@isabella-janssen isabella-janssen mentioned this pull request May 6, 2025
@isabella-janssen isabella-janssen marked this pull request as ready for review May 6, 2025 20:22
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 6, 2025
@openshift-ci openshift-ci bot requested review from cheesesashimi and djoshy May 6, 2025 20:23
@isabella-janssen
Copy link
Member Author

/payload-job-with-prs periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn-techpreview-serial #5034

Copy link
Contributor

openshift-ci bot commented May 6, 2025

@isabella-janssen: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn-techpreview-serial

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/f7107030-2acc-11f0-85d1-2b41fb50677b-0

@isabella-janssen
Copy link
Member Author

/payload-job-with-prs periodic-ci-openshift-release-master-ci-4.19-e2e-azure-ovn-techpreview-serial #5034

Copy link
Contributor

openshift-ci bot commented May 6, 2025

@isabella-janssen: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.19-e2e-azure-ovn-techpreview-serial

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/1e57d3e0-2acd-11f0-8c8e-ee3d28450af1-0

@isabella-janssen
Copy link
Member Author

/payload-job-with-prs periodic-ci-openshift-release-master-ci-4.19-e2e-gcp-ovn-techpreview-serial #5034

Copy link
Contributor

openshift-ci bot commented May 6, 2025

@isabella-janssen: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.19-e2e-gcp-ovn-techpreview-serial

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/29dbd360-2acd-11f0-944c-1a7023fa619a-0

@isabella-janssen
Copy link
Member Author

/payload-job-with-prs periodic-ci-openshift-release-master-nightly-4.19-e2e-vsphere-ovn-techpreview-serial #5034

Copy link
Contributor

openshift-ci bot commented May 6, 2025

@isabella-janssen: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-nightly-4.19-e2e-vsphere-ovn-techpreview-serial

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/3ceb9620-2acd-11f0-8f67-4e7e524cbd0e-0

Copy link
Contributor

openshift-ci bot commented May 7, 2025

@isabella-janssen: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/okd-scos-e2e-aws-ovn 8ba4f20 link false /test okd-scos-e2e-aws-ovn
ci/prow/e2e-azure-ovn-upgrade-out-of-change 8ba4f20 link false /test e2e-azure-ovn-upgrade-out-of-change
ci/prow/e2e-gcp-op 8ba4f20 link true /test e2e-gcp-op

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@isabella-janssen
Copy link
Member Author

/payload-job periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn-techpreview-serial periodic-ci-openshift-release-master-ci-4.19-e2e-azure-ovn-techpreview-serial periodic-ci-openshift-release-master-ci-4.19-e2e-gcp-ovn-techpreview-serial periodic-ci-openshift-release-master-nightly-4.19-e2e-vsphere-ovn-techpreview-serial

Copy link
Contributor

openshift-ci bot commented May 7, 2025

@isabella-janssen: trigger 4 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn-techpreview-serial
  • periodic-ci-openshift-release-master-ci-4.19-e2e-azure-ovn-techpreview-serial
  • periodic-ci-openshift-release-master-ci-4.19-e2e-gcp-ovn-techpreview-serial
  • periodic-ci-openshift-release-master-nightly-4.19-e2e-vsphere-ovn-techpreview-serial

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/6002f400-2b36-11f0-834f-819f1d385f81-0

@isabella-janssen
Copy link
Member Author

/retest-required

@djoshy
Copy link
Contributor

djoshy commented May 7, 2025

/lgtm

Nice work debugging this! 😄

/hold for QE but feel free to unhold if not required

@openshift-ci openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 7, 2025
@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label May 7, 2025
@pablintino
Copy link
Contributor

I do not have all the required context but I'd say lgtm based on the Jira card

@ptalgulk01
Copy link

Pre-merge verification steps:
Verified using IPI based AWS cluster

launch 4.20,openshift/machine-config-operator#5023 aws,techpreview
  1. Create the Custom MCP
Example
$ oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: infra
spec:
  machineConfigSelector:
    matchExpressions:
      - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]}
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/infra: ""
EOF
$ oc label node/ip-10-0-108-115.us-east-2.compute.internal node-role.kubernetes.io/infra=
2. Apply Valid PIS
Example
oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
  labels:
    machineconfiguration.openshift.io/role: infra
  name: pinned-images
spec:
  pinnedImages:
  - name: "quay.io/openshifttest/busybox@sha256:0415f56ccc05526f2af5a7ae8654baec97d4a614f24736e8eef41a4591f08019"
  - name: quay.io/openshifttest/alpine@sha256:be92b18a369e989a6e86ac840b7f23ce0052467de551b064796d67280dfa06d5
EOF
3. Check the PIS status via MCN
$ oc describe machineconfignode ip-10-0-108-115.us-east-2.compute.internal
    Last Transition Time:  2025-05-07T16:34:18Z
    Message:               All pinned image sets complete
    Reason:                AsExpected
    Status:                False
    Type:                  PinnedImageSetsProgressing
.....
    Last Transition Time:  2025-05-07T16:34:18Z
    Message:               All is good
    Reason:                AsExpected
    Status:                False
    Type:                  PinnedImageSetsDegraded
.....
  Pinned Image Sets:
    Current Generation:  1
    Desired Generation:  1
    Name:                pinned-images
4. Apply Invalid PIS
Example
oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
  labels:
    machineconfiguration.openshift.io/role: infra
  name: pinned-bad-image
spec:
  pinnedImages:
  - name: "quay.io/openshiftfake/fakeimage@sha256:0415f56ccc05526f2af5a7ae8654baec97d4a614f24736e8eef41a4591f08019"
EOF
5. Check the PIS status via MCN
oc describe machineconfignode ip-10-0-108-115.us-east-2.compute.internal | tail -n 10
....
    Last Transition Time:  2025-05-07T16:44:28Z
    Message:               node is prefetching images: ip-10-0-108-115.us-east-2.compute.internal
    Reason:                ImagePrefetch
    Status:                True
    Type:                  PinnedImageSetsProgressing
.....
    Last Transition Time:  2025-05-07T16:44:28Z
    Message:               One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details
    Reason:                PrefetchFailed
    Status:                True
    Type:                  PinnedImageSetsDegraded
.....    
  Config Version:
    Current:            rendered-infra-bbb24207a4af065d3c0b012e1c58d912
    Desired:            rendered-infra-bbb24207a4af065d3c0b012e1c58d912
  Observed Generation:  5
  Pinned Image Sets:
    Desired Generation:            1
    Last Failed Generation:        1
    Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/openshiftfake/fakeimage@sha256:0415f56ccc05526f2af5a7ae8654baec97d4a614f24736e8eef41a4591f08019": exit status 125
    Name:                          pinned-bad-image
Events:                            

/label qe-approved

@openshift-ci openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label May 7, 2025
@isabella-janssen
Copy link
Member Author

/payload-job periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn-techpreview-serial periodic-ci-openshift-release-master-ci-4.19-e2e-azure-ovn-techpreview-serial periodic-ci-openshift-release-master-ci-4.19-e2e-gcp-ovn-techpreview-serial periodic-ci-openshift-release-master-nightly-4.19-e2e-vsphere-ovn-techpreview-serial

Copy link
Contributor

openshift-ci bot commented May 7, 2025

@isabella-janssen: trigger 4 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.19-e2e-aws-ovn-techpreview-serial
  • periodic-ci-openshift-release-master-ci-4.19-e2e-azure-ovn-techpreview-serial
  • periodic-ci-openshift-release-master-ci-4.19-e2e-gcp-ovn-techpreview-serial
  • periodic-ci-openshift-release-master-nightly-4.19-e2e-vsphere-ovn-techpreview-serial

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/2e78d050-2b6b-11f0-8d07-4614b6dbccbb-0

Copy link
Contributor

@yuqi-zhang yuqi-zhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logically lgtm.

Interestingly, the payload run has some failures with some MCO related issues, with the vsphere one failing on Should properly transition through MCN conditions on rebootless node update and the AWS failing on MCO related events. They don't seem related to this PR, but are we concerned about those?

Copy link
Contributor

openshift-ci bot commented May 8, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy, isabella-janssen, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [djoshy,isabella-janssen,yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@isabella-janssen
Copy link
Member Author

@yuqi-zhang Yeah, I saw those failures yesterday in the morning and decided to run the payload tests again to see if it's consistent or if it seemed like a flake. What's most concerning to me is the AWS blocking job failing on events should not repeat pathologically for ns/openshift-machine-config-operator, as I have not seen that issue before and it seems very MCO-related.

That being said, none of the issues seem like they would be results of changes in this PR. 🤔

@isabella-janssen
Copy link
Member Author

/payload-job periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview-serial periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-techpreview periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-techpreview-serial periodic-ci-openshift-release-master-nightly-4.20-e2e-vsphere-ovn-techpreview-serial

Copy link
Contributor

openshift-ci bot commented May 8, 2025

@isabella-janssen: trigger 4 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

  • periodic-ci-openshift-release-master-ci-4.20-e2e-aws-ovn-techpreview-serial
  • periodic-ci-openshift-release-master-ci-4.20-e2e-azure-ovn-techpreview
  • periodic-ci-openshift-release-master-ci-4.20-e2e-gcp-ovn-techpreview-serial
  • periodic-ci-openshift-release-master-nightly-4.20-e2e-vsphere-ovn-techpreview-serial

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/93780a00-2c0d-11f0-9626-d0146da4ac88-0

@isabella-janssen
Copy link
Member Author

/unhold

The payload rehearsal tests against the 4.20 periodic jobs looked good. None of the failures related to PIS or MCN, so this PR should be safe to merge.

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 8, 2025
@openshift-merge-bot openshift-merge-bot bot merged commit 647e51e into openshift:main May 8, 2025
16 of 18 checks passed
@openshift-ci-robot
Copy link
Contributor

@isabella-janssen: Jira Issue OCPBUGS-55450: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-55450 has been moved to the MODIFIED state.

In response to this:

Closes: OCPBUGS-55450

- What I did
This adds a check to only set Status.PinnedImageSets.CurrentGeneration in a MachineConfigNode object when the generation value is valid, or non-zero.

- How to verify it

  1. Launch a cluster with this PR.
  2. Apply an invalid PIS.
Example Invalid PIS targeting "custom" MCP
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "custom"
spec:
 pinnedImages:
  - name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86d26e7ebcccd6f07a75db5b1e56283b25c2ee1c6a755d6ffc5a4d59beb9cdef
3. Check that the PIS reference in the MCN has values for `Name`, `DesiredGeneration`, `LastFailedGeneration`, and `LastFailedGenerationError`.
...
Status:
 Conditions:
   Last Transition Time:  2025-05-07T13:11:46Z
   Message:               node is prefetching images: ip-10-0-46-80.ec2.internal
   Reason:                ImagePrefetch
   Status:                True
   Type:                  PinnedImageSetsProgressing
...
   Last Transition Time:  2025-05-07T13:11:48Z
   Message:               One or more PinnedImageSet is experiencing an error. See PinnedImageSet list for more details
   Reason:                PrefetchFailed
   Status:                True
   Type:                  PinnedImageSetsDegraded
...
Pinned Image Sets:
   Desired Generation:            1
   Last Failed Generation:        1
   Last Failed Generation Error:  failed to execute podman manifest inspect for "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:86d26e7ebcccd6f07a75db5b1e56283b25c2ee1c6a755d6ffc5a4d59beb9cdef": exit status 125
   Name:                          test-pinned
  1. Apply a valid PIS.
Example valid PIS targeting "custom" MCP
apiVersion: machineconfiguration.openshift.io/v1
kind: PinnedImageSet
metadata:
 name: test-pinned
 labels:
   machineconfiguration.openshift.io/role: "custom"
spec:
 pinnedImages:
  - name: quay.io/openshifttest/busybox@sha256:c5439d7db88ab5423999530349d327b04279ad3161d7596d2126dfb5b02bfd1
  1. Check that the PIS reference in the MCN has values for Name, DesiredGeneration, and CurrentGeneration.
...
Status:
 Conditions:
   Last Transition Time:  2025-05-07T13:21:19Z
   Message:               All pinned image sets complete
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsProgressing
...
Last Transition Time:  2025-05-07T13:21:19Z
   Message:               All is good
   Reason:                AsExpected
   Status:                False
   Type:                  PinnedImageSetsDegraded
...
 Pinned Image Sets:
   Current Generation:  2
   Desired Generation:  2
   Name:                test-pinned
  1. Check that there are no error logs (especially those starting with "Error applying MCN status").

Note: To see the original failures, see this example payload job: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-machine-config-operator-5034-ci-4.19-e2e-gcp-ovn-techpreview-serial/1919889377152995328.

- Description for the changelog
OCPBUGS-55450: Only populate Status.PinnedImageSets.CurrentGeneration on valid generation value

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@isabella-janssen isabella-janssen deleted the ocpbugs-55450 branch May 8, 2025 19:11
@isabella-janssen
Copy link
Member Author

/cherry-pick release-4.19

@openshift-cherrypick-robot

@isabella-janssen: new pull request created: #5039

In response to this:

/cherry-pick release-4.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. qe-approved Signifies that QE has signed off on this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants