ci: configure retries on tests #20891
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kuisathaverat
added
automation
ci
Team:Automation
botelastic
bot
added
needs_team
mdelapenya
approved these changes
Sep 1, 2020
cachedout
approved these changes
Sep 1, 2020
Co-authored-by: cachedout <mike.place@elastic.co>
💔 Tests Failed
Expand to view the summary
Build stats
Test stats 🧪
Test errorsExpand to view the tests failures
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
|
+1 to add retries, but only if we have also something to track flaky tests. If we only add retries we may end-up with more and worse flaky tests, as CI will be less sensible to flakiness. Worst case we may end-up with tests that only sometimes pass and 3 retries won't be enough. Would this plugin also allow to generate some report on tests that need more retries so we can still act on them? Maybe another option is to use the |
Well, I agree, but 3 runs are not much and this will detect really painful tests we should fix on the first instance, after fix 'em all, we would move the retry to 2, and so on. In that way, we will remove all flaky test and we add a layer of resilience to infrastructure issues.
no, it does not support to report the number of runs right now, see pytest-dev/pytest-rerunfailures#97
It is possible to rerun only flaky |
kuisathaverat
commented
Sep 7, 2020
|
finally, we introduced the plugin and the environment variable empty, so you have to enable retries manually. The plugin allows us to mark tests as flaky and allow retry them. |
💔 Tests Failed
Expand to view the summary
Build stats
Test stats 🧪
Test errorsExpand to view the tests failures
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
v1v
pushed a commit
to v1v/beats
that referenced
this pull request
Sep 24, 2020
v1v
added a commit
that referenced
this pull request
Sep 28, 2020
v1v
added a commit
to v1v/beats
that referenced
this pull request
Sep 28, 2020
* upstream/master: Skip flaky test TestClientPublishEventKerberosAware (elastic#21296) [CI] Enable macOS builds for branches/tags (elastic#21323) [CI] refactor runbld post build action (elastic#21256) [CI] cherry-pick elastic#20891 (elastic#21314)
v1v
added a commit
to v1v/beats
that referenced
this pull request
Oct 7, 2020
Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>
v1v
added a commit
to v1v/beats
that referenced
this pull request
Oct 7, 2020
Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>
v1v
added a commit
that referenced
this pull request
Oct 7, 2020
v1v
added a commit
that referenced
this pull request
Oct 7, 2020
melchiormoulin
pushed a commit
to melchiormoulin/beats
that referenced
this pull request
Oct 14, 2020
* feat: configure retries on tests * Update Jenkinsfile Co-authored-by: cachedout <mike.place@elastic.co> * Update Jenkinsfile Co-authored-by: cachedout <mike.place@elastic.co>
kuisathaverat
added a commit
to kuisathaverat/beats
that referenced
this pull request
Oct 20, 2020
* feat: configure retries on tests * Update Jenkinsfile Co-authored-by: cachedout <mike.place@elastic.co> * Update Jenkinsfile Co-authored-by: cachedout <mike.place@elastic.co> # Conflicts: # Jenkinsfile
kuisathaverat
added a commit
to kuisathaverat/beats
that referenced
this pull request
Oct 20, 2020
* feat: configure retries on tests * Update Jenkinsfile Co-authored-by: cachedout <mike.place@elastic.co> * Update Jenkinsfile Co-authored-by: cachedout <mike.place@elastic.co> # Conflicts: # Jenkinsfile
kuisathaverat
added a commit
that referenced
this pull request
Oct 20, 2020
kuisathaverat
added a commit
that referenced
this pull request
Oct 20, 2020
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com> Co-authored-by: Ivan Fernandez Calvo <kuisathaverat@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
it adds support for retry test with pyTests. (https://github.com/pytest-dev/pytest-rerunfailures)
Why is it important?
some test fails occasionally but there is not really an error, because of that retries the failed test 3 times with a little sleep of 15 seconds will stabilize the test results on the integration test.
Checklist
[ ] My code follows the style guidelines of this project[ ] I have commented my code, particularly in hard-to-understand areas[ ] I have made corresponding changes to the documentation[ ] I have made corresponding change to the default configuration files[ ] I have added tests that prove my fix is effective or that my feature works[ ] I have added an entry inCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.