Add module loading performance check to CI #5418
Conversation
|
View a preview at https://prompt.ws/r/Azure/azure-cli/5418 |
| @@ -3,6 +3,8 @@ | |||
| # Licensed under the MIT License. See License.txt in the project root for license information. | |||
| # -------------------------------------------------------------------------------------------- | |||
|
|
|||
| from msrestazure.tools import resource_id | |||
There was a problem hiding this comment.
This is a perfomance regressions introduced to ensure the test fails.
There was a problem hiding this comment.
Reverted after confirming this triggered the fail.
| @@ -7,6 +7,8 @@ | |||
| import os.path | |||
| import re | |||
|
|
|||
| from azure.mgmt.network.models import Subnet | |||
There was a problem hiding this comment.
Another performance hit that should fail.
There was a problem hiding this comment.
Reverted after confirming this triggered a fail.
tjprescott
changed the title
| @@ -45,51 +51,76 @@ def pstdev(data): | |||
| ss = sq_deviation(data) | |||
| return (ss/n) ** 0.5 | |||
|
|
|||
|
|
|||
| def print_values(data): | |||
| print('{:<20} {:>12} {:>12} {:>12} {:>25}'.format('Module', 'Average', 'Threshold', 'Stdev', 'Values')) | |||
| 'batch': 25, | ||
| 'storage': 50, | ||
| TOTAL: 300 | ||
| } |
There was a problem hiding this comment.
It would be nice if these values can be configured through environment variable.
There was a problem hiding this comment.
If these values end up being problematic, then I'd have to use a more sophisticated method to essentially determine a scalable load-time profile and detect deviations from that. However, since we are always running these on a stable build, my experience (and hope!) is that we won't see wild variations.
Closes #5326.
This PR runs
azdev verify module-load-perfon a separate stage to detect and prevent significant performance regressions.I will remove the DNM label once I have done the following: