Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Test] SearchOperationTest is flaky #3426

Closed
4 tasks
peternied opened this issue Sep 29, 2023 · 6 comments
Closed
4 tasks

[Test] SearchOperationTest is flaky #3426

peternied opened this issue Sep 29, 2023 · 6 comments
Assignees
@MaciejMierzwa
Labels
flaky-test Flaky Test issue good first issue These are recommended starting points for newcomers looking to make their first contributions. triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable.

Comments

@peternied
Copy link
Member

peternied commented Sep 29, 2023 •
edited
Loading

Description

  • There are many tests that use audit log validation of scenarios and it seems either the number of requests is not always consistent
  • There are many tests that use reindexing and it looks like the operation isn't being confirmed as completed before the tests continue
  • There are many tests that use alias creation / clean up and it seems resolution on these actions is not completing before the tests continue causing issues.

Repo

  1. Go to src/integrationTest/java/org/opensearch/security/SearchOperationTest.java
  2. Find the @Ignore many tests in this class
  3. Delete that attribute
  4. Run the tests with ./gradlew integrationTest --tests org.opensearch.security.SearchOperationTest

Exit Criteria

  • Find the root cause(s) of the audit log mismatches, all audit log data should expect exact counts, or have a deterministic function to calculate the count based on number of requests, etc.
  • Find the root cause of the reindexing issues, add a deterministic process to be sure this has completed, note; this could involve a product fix
  • Find the root cause of the aliases issues, add a deterministic process to be sure this has completed, note; this could involve a product fix
  • Run the test at least 10 times and see no failures - how to do this
@peternied peternied added the flaky-test Flaky Test issue label Sep 29, 2023
@github-actions github-actions bot added the untriaged Require the attention of the repository maintainers and may need to be prioritized label Sep 29, 2023
@davidlago davidlago added good first issue These are recommended starting points for newcomers looking to make their first contributions. triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable. and removed untriaged Require the attention of the repository maintainers and may need to be prioritized labels Oct 2, 2023
@MaciejMierzwa
Copy link
Contributor

Looking into it

@MaciejMierzwa
Copy link
Contributor

MaciejMierzwa commented Oct 13, 2023 •
edited
Loading

This issue is linked to this one: #2141
Looking into both, the 2141 one is still open but somebody assigned issue to themself.

@MaciejMierzwa MaciejMierzwa mentioned this issue Oct 26, 2023
Merged
@MaciejMierzwa
Copy link
Contributor

Here's PR: #3602
Seems like issue is similar to this one #1917.

@MaciejMierzwa MaciejMierzwa mentioned this issue Nov 9, 2023
Open
@stephen-crawford
Copy link
Contributor

This is the PR that resolves this issue: #3602

stephen-crawford pushed a commit that referenced this issue Dec 14, 2023
@MaciejMierzwa
Search operation test flakiness fix (#3602)
9da4a78 
### Description
test PR 
### Issues Resolved
- #3426
- #2141
- #2169

Similar to this task:
#1917
From what I've noticed some logs are duplicated on faster machines.
During test creation audit logging results were added based on actual
results produced by tests. Now if there are slower moments where logging
produces non-duplicated logs, those were marked as failed.

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

---------

Signed-off-by: Maciej Mierzwa <dev.maciej.mierzwa@gmail.com>
Signed-off-by: MaciejMierzwa <dev.maciej.mierzwa@gmail.com>
MaciejMierzwa added a commit to MaciejMierzwa/security that referenced this issue Dec 18, 2023
@MaciejMierzwa
Search operation test flakiness fix (opensearch-project#3602)
4962695 
Search operation test flakiness fix
- opensearch-project#3426
- opensearch-project#2141
- opensearch-project#2169

Similar to this task:
opensearch-project#1917
From what I've noticed some logs are duplicated on faster machines.
During test creation audit logging results were added based on actual
results produced by tests. Now if there are slower moments where logging
produces non-duplicated logs, those were marked as failed.

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

---------

Signed-off-by: Maciej Mierzwa <dev.maciej.mierzwa@gmail.com>
Signed-off-by: MaciejMierzwa <dev.maciej.mierzwa@gmail.com>
(cherry picked from commit 9da4a78)
@MaciejMierzwa MaciejMierzwa mentioned this issue Dec 18, 2023
Merged
MaciejMierzwa added a commit to MaciejMierzwa/security that referenced this issue Dec 18, 2023
@MaciejMierzwa
Search operation test flakiness fix (opensearch-project#3602)
3679d43 
Search operation test flakiness fix
- opensearch-project#3426
- opensearch-project#2141
- opensearch-project#2169

Similar to this task:
opensearch-project#1917
From what I've noticed some logs are duplicated on faster machines.
During test creation audit logging results were added based on actual
results produced by tests. Now if there are slower moments where logging
produces non-duplicated logs, those were marked as failed.

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

---------

Signed-off-by: Maciej Mierzwa <dev.maciej.mierzwa@gmail.com>
Signed-off-by: MaciejMierzwa <dev.maciej.mierzwa@gmail.com>
(cherry picked from commit 9da4a78)
Signed-off-by: Maciej Mierzwa <dev.maciej.mierzwa@gmail.com>
MaciejMierzwa added a commit to MaciejMierzwa/security that referenced this issue Dec 18, 2023
@MaciejMierzwa
Search operation test flakiness fix (opensearch-project#3602)
e0eefae 
Search operation test flakiness fix
- opensearch-project#3426
- opensearch-project#2141
- opensearch-project#2169

Similar to this task:
opensearch-project#1917
From what I've noticed some logs are duplicated on faster machines.
During test creation audit logging results were added based on actual
results produced by tests. Now if there are slower moments where logging
produces non-duplicated logs, those were marked as failed.

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

---------

Signed-off-by: Maciej Mierzwa <dev.maciej.mierzwa@gmail.com>
Signed-off-by: MaciejMierzwa <dev.maciej.mierzwa@gmail.com>
(cherry picked from commit 9da4a78)
Signed-off-by: Maciej Mierzwa <dev.maciej.mierzwa@gmail.com>
@MaciejMierzwa
Copy link
Contributor

Hi, here is backport PR: #3862
@scrawfor99 @peternied @cwperks @davidlago @DarshitChanpura

stephen-crawford pushed a commit that referenced this issue Dec 19, 2023
@MaciejMierzwa
Backport/backport 3602 to 2.x (#3862)
45917e9 
### Description
Integration test PR. I followed directions from
#3602 (comment)
not sure if it should look like that though.

### Issues Resolved
This is backport
#3426
#3602

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

Signed-off-by: Maciej Mierzwa <dev.maciej.mierzwa@gmail.com>
Signed-off-by: MaciejMierzwa <dev.maciej.mierzwa@gmail.com>
@cwperks
Copy link
Member

cwperks commented Dec 19, 2023

Closing this issue as associated PRs have been merged.

@cwperks cwperks closed this as completed Dec 19, 2023
prabhask5 pushed a commit to prabhask5/opensearch-security that referenced this issue Jan 11, 2024
@MaciejMierzwa @prabhask5
Search operation test flakiness fix (opensearch-project#3602)
8a76084 
### Description
test PR
### Issues Resolved
- opensearch-project#3426
- opensearch-project#2141
- opensearch-project#2169

Similar to this task:
opensearch-project#1917
From what I've noticed some logs are duplicated on faster machines.
During test creation audit logging results were added based on actual
results produced by tests. Now if there are slower moments where logging
produces non-duplicated logs, those were marked as failed.

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

---------

Signed-off-by: Maciej Mierzwa <dev.maciej.mierzwa@gmail.com>
Signed-off-by: MaciejMierzwa <dev.maciej.mierzwa@gmail.com>
Signed-off-by: Prabhas Kurapati <prabhask@berkeley.edu>
dlin2028 pushed a commit to dlin2028/security that referenced this issue May 1, 2024
@MaciejMierzwa @dlin2028
Search operation test flakiness fix (opensearch-project#3602)
1f14a78 
### Description
test PR 
### Issues Resolved
- opensearch-project#3426
- opensearch-project#2141
- opensearch-project#2169

Similar to this task:
opensearch-project#1917
From what I've noticed some logs are duplicated on faster machines.
During test creation audit logging results were added based on actual
results produced by tests. Now if there are slower moments where logging
produces non-duplicated logs, those were marked as failed.

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

---------

Signed-off-by: Maciej Mierzwa <dev.maciej.mierzwa@gmail.com>
Signed-off-by: MaciejMierzwa <dev.maciej.mierzwa@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MaciejMierzwa MaciejMierzwa

Labels
flaky-test Flaky Test issue good first issue These are recommended starting points for newcomers looking to make their first contributions. triaged Issues labeled as 'Triaged' have been reviewed and are deemed actionable.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@peternied @davidlago @cwperks @stephen-crawford @MaciejMierzwa