Batch generation

[Pan-Qi]

Description

Batch generation pipeline
Currently the generated branch won't be merged until we have decided the release policy.
If the pipeline is failed, notification will be sent to Xiaogang, Yabo, Yeming and me.
This is the successful build, the failed tests are looks expected and need to be fixed through other PRs.
This is the generated branch, which includes both src/ and generated/ folders.

As an example, this build is successfully triggered by test upstream pipelineA on completion. Batch generation will only be triggered when upstream publish is finished successfully.
After test, this pipeline will be changed to be trigger by pipeline on successful completion.

Files description:

.azure-pipelines\
    |__ batch-generation.yml - pipeline definition
    |__ PipelineSteps/BatchGeneration\
        |__ FailedJobTeamsTemplate.html - template for failed build teams' notification message
        |__ analyse-modules.ps1 - analyse module script, include adding module analyse result into AnalyseReport.json
        |__ bath-generate-modules.ps1 - regenerate allocated modules/sub-modules. write log into generate log and add result into GenerationReport
        |__ build-modules.ps1 - build allocated modules and write result into BuildReport
        |__ create-branch.ps1 - script for creating git branch(for batch generation branch)
        |__ filter.ps1 - filter all modules that changed by regeneration and dynamically generate build/analyse/test job matrix by max parallel jobs
        |__ notify-failed-jobs.ps1 - teams message failed build to notify receivers
        |__ prepare.ps1 - filter out all v4 modules, and dynamically generate regeneration job matrix based on max parallel jobs
        |__ report.ps1 - filter out all report files generated by regenerate/build/test jobs, analyse all reports and exit 1 if any report item is failed
        |__ test-modules.ps1 - test sub-modules if different test environment(windows/linux/mac), write test result into TestReport
        |__ util.psm1 - some common functions
    |__ util\
        batch-generation-test-job.yml - yml definition for test jobs

Workflow description

Stages: Generate -> Build -> Test -> Report -> FailedJobNotification

Jobs and steps in each stage: 
Generate 
    |__ Create Batch Generation Branch
        |__ Checkout...
        |__ Get Github PAT from key value
        |__ Create Batch Generation Branch (File to execute: create-branch.ps1)
        |__ Get latest Autorest Version
        |__ Generate Build Targets Matrix (File to execute: prepare.ps1)
        |__ Upload generated targets (job matrix is too large and exceed parameter length limit, so that in parameter only keep matrix keys, and put all the modules to regenerate data in an aritifact file - generationTargets.json. In the following matrix jobs, it will go to fetch exact modules from generationTargets.json artifact with matrix key.)
    |__ Batch Generate:($MatrixKey) - matrix job to regenerate allocated sub-modules
        |__ Chekcout...
        |__ DownloadPipelineArtifact (file to download: generationTargets.json)
        |__ Regenerate Modules (File to execute: batch-generate-modules.ps1)
        |__ Save patch and generation report (Save git patch for regeneration change log and report files while regenerating)
Build
    |__ Filter Changed Modules - Filter out all modules that has code change when regenerate modules by latest version Autorest Powershell
        |__ Checkout...
        |__ Checkout Generation Branch
        |__ DownloadPipelineArtifact - download generationTargets.json to get all filtered V4 modules
        |__ Download all .patch artifacts
        |__ Apply all patches
        |__ Generate Test Targets Matrix - (File to execute: filter.ps1) - Apply all patch files on batch-generation branch, and filter out all changed modules/sub-modules then generate build/analyse/test job matrix based on max parallel jobs
        |__ Upload filtered targets - Same as "Upload generated targets" step, upload matrix file for following matrix jobs
    |__ Build: ($MatrixKey)
        |__ Checkout...
        |__ DownloadPipelineArtifact - download matrix file buildTargets.json
        |__ Build Targets (File to execute: build-modules.ps1)
        |__ Save build artifacts - Save build artifact and BuildReport
Test
    |__ Collect Artifacts - As max parallel jobs for build and test are not same, so the build and test jobs are not identical. So this Job is to collect all build artifacts for following tests reference.
        |__ Download build artifacts - download all modules' build artifacts
        |__ DownloadPipelineArtifact - download all matrix files e.g. analyzeTargets.json/testLinuxTargets.json/...
        |__ Collect modules artifacts - put all artifacts into one folder
        |__ Upload collected artifacts - upload all artifacts into single artifact folder for future reference
    |__ Analyze: ($MatrixKey) - analyze allocated modules and upload analyze result report file
        |__ Download collected artifact to artifacts folder - download all artifacts from filter and build jobs
        |__ Install autorest
        |__ Setup environment for autorest
        |__ Use .NET SDK for Static Analysis
        |__ Analyze modules (File to execute: analyse-modules.ps1)
        |__ Save Analyse Report - upload analyse report
    |__ Test($TestEnv): ($MatrixKey) - test allocated sub-modules in different environments(Windows/Linux/Mac) and upload test result report file
        |__ Checkout...
        |__ Download collected artifact to artifacts folder - download all artifacts from filter and build jobs
        |__ Use .NET SDK for Test
        |__ Test $EnvName - (File to execute: test-modules.ps1)
        |__ Save Test $EnvName Report
Report
    |__ Report - Collect all reports from regenerate/build/analyse/test jobs, and check if there is any error from any module/sub-module, if so, write those failed items and exit 1
        |__ Checkout...
        |__ DownloadPipelineArtifact - download generationTargets.json to know which modules are regenerated
        |__ DownloadPipelineArtifact - download analyseTargets.json/buildTargets.json/testTargets.json to know which modules are built and tested. These 2 steps are intended to confirm there is no failed report file missed.
        |__ Download build artifacts - download all report json files
        |__ Report Results - (File to execute: report.ps1) to check if any report file contains failed record. if so, write failed item and exit 1
        |__ Save Reports - Save collected final report to artifact
FailJobNotification
    |__ FailJobNotification - (File to execute: notify-failed-job.ps1) notify receivers when any job above is failed. 

Mandatory Checklist

• Please choose the target release of Azure PowerShell. (⚠️Target release is a different concept from API readiness. Please click below links for details.)

  • General release
  • Public preview
  • Private preview
  • Engineering build
  • No need for a release

• Check this box to confirm: I have read the Submitting Changes section of CONTRIBUTING.md and reviewed the following information:

• SHOULD update ChangeLog.md file(s) appropriately
  • For SDK-based development mode, update src/{{SERVICE}}/{{SERVICE}}/ChangeLog.md.
    • A snippet outlining the change(s) made in the PR should be written under the ## Upcoming Release header in the past tense.
  • For autorest-based development mode, include the changelog in the PR description.
  • Should not change ChangeLog.md if no new release is required, such as fixing test case only.
• SHOULD regenerate markdown help files if there is cmdlet API change. Instruction
• SHOULD have proper test coverage for changes in pull request.
• SHOULD NOT adjust version of module manually in pull request

[azure-client-tools-bot-prd]

Thanks for your contribution! The pull request validation has started. Please revisit this comment for updated status.

[Copilot]

Pull Request Overview

Adds a batch generation pipeline for Azure PowerShell that parallelizes module regeneration, build, analysis, and testing, and sends failure notifications.

• Introduces utility functions for mapping, grouping, and Teams notifications
• Adds PS scripts to prepare targets, filter changes, generate, build, analyze, test, and report
• Configures Azure Pipelines job matrix in batch-generation-test-job.yml

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
.azure-pipelines/util/batch-generation-test-job.yml	Defines parameterized test job with matrix for Windows agents
.azure-pipelines/PipelineSteps/BatchGeneration/util.psm1	Helper functions: module mapping, grouping, matrix writing, messaging
.azure-pipelines/PipelineSteps/BatchGeneration/prepare.ps1	Prepares and serializes generation targets and matrix variable
.azure-pipelines/PipelineSteps/BatchGeneration/filter.ps1	Filters changed modules/submodules and writes build/analyze/test matrices
.azure-pipelines/PipelineSteps/BatchGeneration/batch-generate-modules.ps1	Executes Autorest regeneration per module/submodule and records change logs
.azure-pipelines/PipelineSteps/BatchGeneration/build-modules.ps1	Builds modules based on matrix key and records results
.azure-pipelines/PipelineSteps/BatchGeneration/analyse-modules.ps1	Runs static analysis steps and aggregates failures
.azure-pipelines/PipelineSteps/BatchGeneration/test-modules.ps1	Runs Pester tests for each submodule and collects results
.azure-pipelines/PipelineSteps/BatchGeneration/report.ps1	Aggregates and writes reports for all pipeline stages
.azure-pipelines/PipelineSteps/BatchGeneration/notify-failed-job.ps1	Sends Teams notification on failure using HTML template
.azure-pipelines/PipelineSteps/BatchGeneration/FailedJobTeamsTemplate.html	HTML template for pipeline failure alerts