Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update checks, creating new ones instead of updating the existing job check #54

Open
r0x0d opened this issue Oct 11, 2022 · 4 comments
Open
Assignees
Labels
type: feature New feature or request

Comments

@r0x0d
Copy link

r0x0d commented Oct 11, 2022

Hello!

In Convert2RHEL, we noticed that the checks for this action are not updating the job itself, but rather, they are creating new ones when the job fails.

We wanted to know if that is the correct behavior, or we are setting something wrong in our usage. We would expect that this action would update the job itself with the status, instead of creating a new one and having a confuse status report under the PR… Currently, we have the job that was executed marked as green, and another check with a red mark related to the same job that ran.

Link for the PR: oamg/convert2rhel#528

The image might not show much, but almost all tests failed in this PR, and if you look in the checks, you will notice that we have it duplicated, one with a green mark because the job finished and one with the red mark because the job failed.
image

@bocekm
Copy link
Contributor

bocekm commented Oct 11, 2022

The current behavior @r0x0d explained has another negative side effect. When opening the PR's Checks tab we see all tests as green/passing even if they in fact failed. This is not only misleading but also we can't re-run just the failed tests. Normally there's a button for that in GitHub if some of the tests are reported as failed but in this case the button is not there as the failed tests don't appear there at all.
Screenshot from 2022-10-11 19-20-57

@zmiklank zmiklank self-assigned this Oct 20, 2022
@zmiklank
Copy link
Collaborator

Hi, thank you for creating the issue.

The current behavior @r0x0d explained has another negative side effect. When opening the PR's Checks tab we see all tests as green/passing even if they in fact failed. This is not only misleading but also we can't re-run just the failed tests. There's a button for that in GitHub if some of the tests are reported as failed.

This is intended. When a test fails, its status is red (failed), but overall result of the job is green (meaning all tests have been run and action finished as was expected). In case of some error in the action the overal result of the action is fail - red.
Question is what is the correct approach here. @phracek, any opinion?

In Convert2RHEL, we noticed that the checks for this action are not updating the job itself, but rather, they are creating new ones when the job fails.

I did not run into this issue when running our actions base on testing-farm-as-a-github-action. I need to take closer look at what has happened in your case.

@zmiklank
Copy link
Collaborator

Hi, I think that the double status is caused by the chained calling of the actions in the convert2rhel repository.
IMO firstly the tmt-tests.yml is called which creates first status (https://github.com/oamg/convert2rhel/blob/9a7f1032dcd7df4b18ae7b8415dfa713a0eecae7/.github/workflows/tmt-tests.yml#L86), which is in this case green, as it end with success - reuse-int-tests.yml is called successfully.
Then the testing-farm-as-github-action create its own status - in this case red, as tests failed.

Could that be it?

@r0x0d
Copy link
Author

r0x0d commented Oct 24, 2022

Hi, @zmiklank! That seems like a very plausible cause… I was wondering if there is no way to tell testing-farm-as-github-action to update an existing check instead of creating a new one.

Not sure how that would work with reusable jobs like this, but, that would be super nice to have.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: feature New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants