Skip to content

mute rules upd #8865

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Sep 6, 2024
Merged

mute rules upd #8865

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d56601b
mute rules
naspirato Sep 6, 2024
73cd76e
mute unmute
naspirato Sep 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 20 additions & 8 deletions .github/config/mute_rules.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
## [How to Mute a test](#how-to-mute)
## How to Mute a test <a id="how-to-mute"></a>

- Through a PR Report
- Open report in PR ![screen](https://storage.yandexcloud.net/ydb-public-images/report_mute.png)
Expand All @@ -16,7 +16,7 @@
* Set the `owner` field to the team name (see the issue for the owner's name). ![image.png](https://storage.yandexcloud.net/ydb-public-images/create_issue.png)
* Open [muted_ya.txt](https://github.com/ydb-platform/ydb/blob/main/.github/config/muted_ya.txt) in a new tab and edit it.
* Copy the line under `Add line to muted_ya.txt` (for example, like in the screenshot, `ydb/core/kqp/ut/query KqpStats.SysViewClientLost`) and add it to [muted_ya.txt](https://github.com/ydb-platform/ydb/blob/main/.github/config/muted_ya.txt).
* Edit the branch for merging, for example, replace `{username}-patch-1` with `{username}/mute`.
* Edit the branch for merging, for example, replace `{username}-patch-1` with `mute/{username}`.
* Create a PR - copy the PR name from the issue name.
* Copy the issue description to the PR, keep the line `Not for changelog (changelog entry is not required)`.
* Take "OK" from member of test owner team in PR
Expand All @@ -25,18 +25,31 @@
* Inform test owner team about new mutes - dm or in public chat (with mention of maintainer of team)
* You are awesome!

## [How to UnMute a test](#how-to-unmute)
## How to UnMute a test <a id="how-to-unmute"></a>
--IN PROGRESS--
* Open [muted_ya.txt](https://github.com/ydb-platform/ydb/blob/main/.github/config/muted_ya.txt)
* Press "Edit file" and delete line of test
* Commit changes (Edit the branch for merging, for example, replace `{username}-patch-1` with `mute/{username}`)
* Edit PR name like "UnMute {testname}"
* Take "OK" from member of test owner team in PR
* Merge
* If test have an issue in [Mute and Un-mute](https://github.com/orgs/ydb-platform/projects/45/views/6?visibleFields=%5B%22Title%22%2C%22Assignees%22%2C%22Status%22%2C126637100%5D) in status "Muted" - Move it to "Unmuted"
* If test have an issue in [Mute and Un-mute](https://github.com/orgs/ydb-platform/projects/45/views/6?visibleFields=%5B%22Title%22%2C%22Assignees%22%2C%22Status%22%2C126637100%5D) in status `Muted` - Move it to `Unmuted`
* Link Issue and Pr (field "Development" in issue and PR)
* Move issue to status `Unmuted`
* You are awesome!

## How to manage muted tests by team <a id="how-to-manage"></a>
--IN PROGRESS--
### Explore your tests stability
>If you want to get more info about stability of your test visit [dashboard](https://datalens.yandex/4un3zdm0zcnyr?tab=wED) (fill field `owner`=`{your_team_name}`)
![image.png](https://storage.yandexcloud.net/ydb-public-images/test_analitycs_1.png)
![image.png](https://storage.yandexcloud.net/ydb-public-images/test_analitycs_2.png)
### Find your muted tests
>Not all muted tests have issue in github project about this , we working on it
* Open project [Mute and Un-mute](https://github.com/orgs/ydb-platform/projects/45/views/6?visibleFields=%5B%22Title%22%2C%22Assignees%22%2C%22Status%22%2C126637100%5D)
* click in label with name of your team, example [link to qp](https://github.com/orgs/ydb-platform/projects/45/views/6?filterQuery=owner%3Aqp) muted tests (cgi `?filterQuery=owner%3Aqp`)
* Open `Mute {testname}` issue
* Perform [How to unmute](#how-to-unmute)

## Flaky Tests

Expand All @@ -47,14 +60,14 @@ The CI duty engineer (in progress) checks flaky tests once a day (only working d
- Open the [Flaky](https://datalens.yandex/4un3zdm0zcnyr) dashboard.
- Perform the sections **[Mute Flaky Test](#mute-flaky)** and **[Test Flaps More - Need to Unmute](#unmute-flaky)** once a day or ondemand

### [Mute Flaky Tests](#mute-flaky)
### Mute Flaky Tests <a id="mute-flaky"></a>

Open the [Flaky](https://datalens.yandex/4un3zdm0zcnyr) dashboard.

- Select today's date.
- Look at the tests in the Mute candidate table.

![image.png](/kikimr/ydb-qa/mute-autotests/.files/image-1.png =800x)
![image.png](https://storage.yandexcloud.net/ydb-public-images/mute_candidate.png)

- Select today's date in the `date_window`.
- Select `days_ago_window = 5` (how many days back from the selected day to calculate statistics). Currently, there are calculations for 1 day and 5 days ago.
Expand All @@ -66,7 +79,7 @@ Open the [Flaky](https://datalens.yandex/4un3zdm0zcnyr) dashboard.
- Perform steps from [How to mute](#how-to-mute)
- You are awesome!

### [Test is no longer flaky - Time to Unmute](#unmute-flaky)
### Test is no longer flaky - Time to Unmute <a id="unmute-flaky"></a>

- Open the [Flaky](https://datalens.yandex/4un3zdm0zcnyr) dashboard.
- Look at the tests in the UNMute candidate table.
Expand All @@ -76,4 +89,3 @@ Open the [Flaky](https://datalens.yandex/4un3zdm0zcnyr) dashboard.
- If the `summary:` column shows `mute <= 3` and `success rate >= 98%` - **it's time to enable the test**.
- Perform steps from [How to Unmute](#how-to-unmute)
- You are awesome!