Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

{CI} Split automation full test and automatic schedule test modules #21976

Merged
merged 33 commits into from
Apr 20, 2022

Conversation

wangzelin007
Copy link
Member

@wangzelin007 wangzelin007 commented Apr 7, 2022

Description

  • Add --pytest-args "--durations=0" to record the time spent on each test.
    image
  • Fix top N slowest test PR: {CI} Fix top N slowest Test #21880
  • For every module not in series_modules, enable group worker in pytest, pytest will spawn a number of workers processes equal to the number of available CPUs, and distribute the tests randomly across them. {CI} enable pytest group worker in module test and full test #22047
  • Split automation full test to N pipeline instances
    • Using 8 instances is the best practice based on repeated testing ( N = 8 )
    • Now we support 3 python versions: 3.6 3.9 3.10, So we have 24 ( 3 * 8 ) instances are running in parallel
    • Because we share the VM pool with the azure-sdk team, so we can't set the number of concurrency arbitrarily
      If you add more instances, some instances will be in a waiting state, and the actual execution time will be longer.
  • Automatic scheduling modules in every pipeline instance
    • write a greedy algorithm to distribute modules to each worker ( In the past we test modules order by their name. It will waste a lot of time )
      For each module, we record their historical execution time. Then we assign the module to an instance with the fewest jobs currently to ensure the execution time of each pipeline instance is the same
    • For the newly added module use module average test time to calculate
    • Support specifying which modules need to test in serial mode
    • Support distributing modules based on the number of pipeline instances( N = 8 )

Before and after optimization:
image
image

Testing Guide

History Notes

[Component Name 1] BREAKING CHANGE: az command a: Make some customer-facing breaking change
[Component Name 2] az command b: Add some customer-facing feature


This checklist is used to make sure that common guidelines for a pull request are followed.

@ghost ghost added Auto-Assign Auto assign by bot CI CI labels Apr 7, 2022
@ghost ghost requested review from yonzhan, jiasli and kairu-ms April 7, 2022 08:27
@ghost ghost assigned wangzelin007 Apr 7, 2022
@ghost ghost added this to the Apr 2022 (2022-04-26) milestone Apr 7, 2022
@yonzhan
Copy link
Collaborator

yonzhan commented Apr 7, 2022

CI

@wangzelin007 wangzelin007 merged commit 9bf6fcf into dev Apr 20, 2022
@wangzelin007 wangzelin007 deleted the split_automation_full_test4 branch April 20, 2022 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Auto-Assign Auto assign by bot CI CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants