Attempting to increase reliability of some Pester tests #265
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We're seeing some inconsistent failures in some of the Pester tests. The hypothesis is that GitHub may need a little bit more time after the creation of objects before performing certain operations on them (like renaming repos), or may need more time after deleting them before it will successfully return a 404 on a successive Get request. To see if this is the case, I'm adding some 1 second sleeps at the point where we've seen test case failures in recent CI runs.
HowardWolosky
added
bug
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run PowerShellForGitHub-CI |
|
Azure Pipelines successfully started running 1 pipeline(s). |
8 tasks
HowardWolosky
added a commit
to HowardWolosky/PowerShellForGitHub
that referenced
this pull request
Jul 19, 2020
This is re-implementing microsoft#265 in a more robust way, thanks to a suggestion from @X-Guardian. Instead of adding in one-off sleeps throughout the test codebase, there is now a new configuration value that will allow us to insert a delay before returning the result of _any_ state change request. This should ideally consistently add reliability throughout the entire test codebase.
10 tasks
HowardWolosky
added a commit
that referenced
this pull request
Jul 19, 2020
This is re-implementing #265 in a more robust way, thanks to a suggestion from @X-Guardian in #267. Instead of adding in one-off sleeps throughout the test codebase, there is now a new configuration value `StateChangeDelaySeconds`) that will allow us to insert a delay before returning the result of _any_ state change request. This should ideally consistently add reliability throughout the entire test codebase.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
We're seeing some inconsistent failures in some of the Pester tests.
The hypothesis is that GitHub may need a little bit more time after the creation of objects before performing certain operations on them (like renaming repos), or may need more time after deleting them before it will successfully return a 404 on a successive Get request.
I have added a number of
Start-Sleep's throughout the test codebase wherever we've encountered inconsistent failures, and that appears to have resolved the problem. We may need to continue to do more of these if additional ones pop up.The duration of the sleep itself is controlled by
$script:defaultSleepSecondsForReliabilitywhich is defined inTests/Common.ps1.Long term, I have opened #267 which poses the idea of switching over to mocking out
Invoke-WebRequestfor the majority of the tests, and instead focus on validating the data that it's sending matches the expected values per the API documentation, and having just a limited number of tests that do actual end-to-end testing.Issues Fixed
Fixes #264
References
n/a
Checklist
You actually ran the code that you just wrote, especially if you did just "one last quick change".Comment-based help added/updated, including examples.Formatters were created for any new types being added.New/changed code continues to support the pipeline.Changes to the manifest file follow the manifest guidance.Relevant usage examples have been added/updated in USAGE.md.If desired, ensure your name is added to our Contributors list