merge main branch (#22421)

* [Release issue status] Reduce max papallel (#22334)

* update assignee

* Update release_issue_status.yml

* [ServiceBus] fixed retry backoff (#22317)

* inital fixed retry commit

* changelog + version

* pylint

* [formrecognizer] Fix test assert for changing description value (#22324)

* fix assert for changing description value

* fix advanced samples file reference

* skip copy model samples

* Update README.md (#22259)

* Update README.md

updated Read me

* Update sdk/textanalytics/azure-ai-textanalytics/samples/README.md

Co-authored-by: Krista Pratico <krpratic@microsoft.com>

Co-authored-by: Krista Pratico <krpratic@microsoft.com>

* enable identity for py27 (#22341)

* enable identity for py27

* update

* drop py27 for ditital twins (#22343)

* [EventHub] drop python 2.7/3.6 (#22319)

* drop 2.7/3.6

* checkpointstoreblobs

* bump versions

* [SchemaRegistry] drop 2.7/36 (#22318)

* drop 2.7/36

* version

* drop 2.7/3.6

* Sync eng/common directory with azure-sdk-tools for PR 2484 (#22346)

* Add weekly pipeline generation to prepare-pipelines template

* Add succeeded condition to pipeline generation pipelines

Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>

* [Key Vault] Skip tests failing with SAS issues (#22351)

* [AutoRelease] t2-consumption-2022-01-06-98359 (#22360)

* CodeGen from PR 17013 in Azure/azure-rest-api-specs
[Hub Generated] Review request for Microsoft.Consumption to add version stable/2021-10-01 (#17013)

* fixing s360 for these

* fixing styling issue

* Adding 204 to examples

* Fixing the model val for reservation by adding proper guid

Co-authored-by: Mita Garg <v-migarg@microsoft.com>

* version,CHANGELOG

* test

* update dev_requirements.txt

* update tests

Co-authored-by: SDKAuto <sdkautomation@microsoft.com>
Co-authored-by: Mita Garg <v-migarg@microsoft.com>
Co-authored-by: PythonSdkPipelines <PythonSdkPipelines>
Co-authored-by: msyyc <70930885+msyyc@users.noreply.github.com>
Co-authored-by: BigCat20196 <1095260342@qq.com>

* [AutoRelease] t2-streamanalytics-2022-01-06-51238 (#22356)

* CodeGen from PR 17260 in Azure/azure-rest-api-specs
Update readme.python.md (#17260)

* version,CHANGELOG

* Update CHANGELOG.md

* Update _version.py

Co-authored-by: SDKAuto <sdkautomation@microsoft.com>
Co-authored-by: PythonSdkPipelines <PythonSdkPipelines>
Co-authored-by: Yuchao Yan <yuchaoyan@microsoft.com>

* t2-netapp-2021-08-01 (#22316)

* CodeGen from PR 16381 in Azure/azure-rest-api-specs
[NetAppFiles] Anf 16015 net app files 2021-08-01 (#16381)

* Add 2021-08-01 folder

* Bump to api-version 2021-08

* Apply cahnges to api-version 2021-08

* Apply changes to api-version 2021-08

* Add VolumeGroups API

* prettier

* Add VolumeGroups API

* Update volumeType desciption

* Remove Name from backupPolicyPropertyBag

* Remove Name from backupPolicy Examples

* version,CHANGELOG

* test

* Sync eng/common directory with azure-sdk-tools for PR 2327 (#22021)

* new recordings

* Update CHANGELOG.md

* merge main

* Update shared_requirements.txt

Co-authored-by: SDKAuto <sdkautomation@microsoft.com>
Co-authored-by: PythonSdkPipelines <PythonSdkPipelines>
Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com>
Co-authored-by: Yuchao Yan <yuchaoyan@microsoft.com>
Co-authored-by: msyyc <70930885+msyyc@users.noreply.github.com>

* [AutoRelease] t2-notificationhubs-2022-01-06-74264 (#22359)

* CodeGen from PR 17259 in Azure/azure-rest-api-specs
Update readme.python.md (#17259)

* version,CHANGELOG

* test

* Update CHANGELOG.md

* Update _version.py

Co-authored-by: SDKAuto <sdkautomation@microsoft.com>
Co-authored-by: PythonSdkPipelines <PythonSdkPipelines>
Co-authored-by: Yuchao Yan <yuchaoyan@microsoft.com>

* [Perf] Fix batch sizes in SampleBatchTest (#22352)

* Handle skipping docker build when PushImages is set and there is no dockerfile (#22107)

Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>

* drop Python 2.7/3.6 (#22320)

* [Key Vault] Drop 2.7, 3.6 support (#22367)

* Fixed the printing message. (#22301)

* synapse artifacts jan updates (#22306)

* synapse artifacts jan updates

* SHA 949388b

* update changelog

* [AutoRelease] t2-compute-2022-01-06-94013 (#22364)

* CodeGen from PR 17254 in Azure/azure-rest-api-specs
Compute add batch tag package-2021-08-01 (#17254)

* version,CHANGELOG

* test

Co-authored-by: SDKAuto <sdkautomation@microsoft.com>
Co-authored-by: PythonSdkPipelines <PythonSdkPipelines>
Co-authored-by: Yuchao Yan <yuchaoyan@microsoft.com>

* [ServiceBus&EventHubs] remove RetryMode enum (#22369)

* remove RetryMode from SB

* remove RetryMode from EH

* annas comments

* delete web2021-02-01 (#22375)

* Hotfix for pipeline python version 2.7/3.6 task issues (#22363)

* excluded pypy3 tests run to avoid pipeline failure due to dropping support 2.7/3.6

* msrest version fixed for sms&phonenumber packages

* azure-mgmt-core version fixed for azure-mgmt-communication package

* analyzing task fail fix

* excluded 2.7/3.6 tests run to avoid pipeline failure due to dropping support 2.7/3.6

* removed unneccessary config from test.yml

* azure-mgmt-core version reverted for azure-mgmt-communication

* fixed dependency conflicts

Co-authored-by: Aigerim Beishenbekova <aigerimb@DESKTOP-2660OE9.corp.microsoft.com>
Co-authored-by: Aigerim Beishenbekova <aigerimb@DESKTOP-TLPRG45.reddog.microsoft.com>

* update to uamqp 1.5.0 (#22348)

* revert the change of bumping dep version (#22382)

* [textanalytics] run pyupgrade on textanalytics (#22344)

* run pyupgrade on samples

* run pyupgrade on tests

* run pyupgrade on src code

* fix mock import and pylint

* [KV] Remove metapackage from ci.yml (#22385)

* Update README.md (#22345)

* Remove py27 and py36 from all test matrices. Leave py27 testing enabled for core. (#22388)

* remove py27 and py36 from all test matrices. leave py27 testing enabled for core

* [ServiceBus] expand kwargs in public API (#22353)

* inital expand kwargs w/o mgmt models

* fix mypy/pylint

* mypy

* adam + anna comments

* lint

* lint bad whitespace

* remove retry

* adams comments

* lint

* Update text analytics live test config to support sovereign cloud testing (#22376)

* [metricsadvisor] convert to test proxy (#21974)

* update sync base testcases and conftest to use test proxy

* remove dep on unittest

* add sanitizers for test secrets

* convert one test

* fix weird spacing

* refactor base testcase into preparer

* convert test_alert_config.py + recordings

* recordings

* updates for aad tests

* test_credential and test_credential_entities, no recordings b/c need to fix sanitizer

* add rstrip / to client endpoint

* convert data_feed_ingestion

* convert data_feeds

* convert detection_config

* convert hooks

* update ma client tests

* refactor to simplify base test class and secrets needed

* refactor, parametrize auth option

* refactor / parametrize async + add preparer

* convert async tests to proxy

* add oauth sanitizer

* add test proxy to ci.yaml

* update testcases and have proxy auto start

* rename test_id -> ids, fix clean up so we raise better errors, add more randomization in names

* rerecord and delete old recordings

* rename tests to fix max path length and try removing max parallel for live tests

* ugh fix for py2.7

* fix envars for live tests

* fixes for live tests

* adjust list data feeds tests to be less flaky

* add rstrip before sanitizing

* Sync eng/common directory with azure-sdk-tools for PR 2500 (#22395)

* Update pipeline generator tool feed to azure-sdk-for-net

* Update pipeline generator tool version

Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>

* add assignee (#22400)

* [metricsadvisor] shorten recording names (#22412)

* shorten recording names

* rename the sync recordings

* release failed because release date is too far away. (#22413)

* disable detect api changes RE #22416 (#22417)

Co-authored-by: Yuchao Yan <yuchaoyan@microsoft.com>
Co-authored-by: swathipil <76007337+swathipil@users.noreply.github.com>
Co-authored-by: catalinaperalta <catalinaperaltah@hotmail.com>
Co-authored-by: Baher Abdullah <86964919+BaherAbdullah@users.noreply.github.com>
Co-authored-by: Krista Pratico <krpratic@microsoft.com>
Co-authored-by: Xiang Yan <xiangsjtu@gmail.com>
Co-authored-by: Azure SDK Bot <53356347+azure-sdk@users.noreply.github.com>
Co-authored-by: Ben Broderick Phillips <bebroder@microsoft.com>
Co-authored-by: McCoy Patiño <39780829+mccoyp@users.noreply.github.com>
Co-authored-by: Azure CLI Bot <azpycli@microsoft.com>
Co-authored-by: SDKAuto <sdkautomation@microsoft.com>
Co-authored-by: Mita Garg <v-migarg@microsoft.com>
Co-authored-by: msyyc <70930885+msyyc@users.noreply.github.com>
Co-authored-by: BigCat20196 <1095260342@qq.com>
Co-authored-by: Hödd <hodd@netapp.com>
Co-authored-by: Mike Harder <mharder@microsoft.com>
Co-authored-by: Sima Zhu <48036328+sima-zhu@users.noreply.github.com>
Co-authored-by: Jiefeng Chen <51037443+BigCat20196@users.noreply.github.com>
Co-authored-by: Aigerim <aykobb@gmail.com>
Co-authored-by: Aigerim Beishenbekova <aigerimb@DESKTOP-2660OE9.corp.microsoft.com>
Co-authored-by: Aigerim Beishenbekova <aigerimb@DESKTOP-TLPRG45.reddog.microsoft.com>
Co-authored-by: Adam Ling (MSFT) <adam_ling@outlook.com>
Co-authored-by: Scott Beddall <45376673+scbedd@users.noreply.github.com>
Co-authored-by: giakas <giakas@microsoft.com>

README.md

@@ -6,7 +6,7 @@ This repository is for active development of the Azure SDK for Python. For consu
 
 ## _Disclaimer_
 
-_Azure SDK Python packages support for Python 2.7 is ending 01 January 2022. For more information and questions, please refer to https://github.com/Azure/azure-sdk-for-python/issues/20691_
+_Azure SDK Python packages support for Python 2.7 has ended 01 January 2022. For more information and questions, please refer to https://github.com/Azure/azure-sdk-for-python/issues/20691_
 
 ## Getting started
 

eng/common/pipelines/templates/steps/install-pipeline-generation.yml

@@ -9,8 +9,8 @@ steps:
   - script: >
       dotnet tool install
       Azure.Sdk.Tools.PipelineGenerator
-      --version 1.0.2-dev.20210621.4
-      --add-source https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk/nuget/v3/index.json
+      --version 1.0.2-dev.20220106.2
+      --add-source https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json
       --tool-path ${{parameters.ToolPath}}
     workingDirectory: $(Pipeline.Workspace)/pipeline-generator
     displayName: 'Install pipeline generator tool'

eng/common/pipelines/templates/steps/prepare-pipelines.yml

@@ -71,7 +71,26 @@ steps:
         --debug
         ${{parameters.TestsConventionOptions}}
       displayName: Create Live Test pipelines for public repository
-      condition: ne('${{parameters.TestsConventionOptions}}','')
+      condition: and(succeeded(), ne('${{parameters.TestsConventionOptions}}',''))
+      env:
+        PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat)
+    - script: >
+        $(Pipeline.Workspace)/pipeline-generator/pipeline-generator
+        --organization https://dev.azure.com/azure-sdk
+        --project internal
+        --prefix ${{parameters.Prefix}}
+        --devopspath "\${{parameters.Prefix}}"
+        --path $(System.DefaultWorkingDirectory)/sdk
+        --endpoint Azure
+        --repository ${{parameters.Repository}}
+        --convention weekly
+        --agentpool Hosted
+        --branch refs/heads/$(DefaultBranch)
+        --patvar PATVAR
+        --debug
+        ${{parameters.TestsConventionOptions}}
+      displayName: Create Weekly (Multi-Cloud) Live Test pipelines for public repository
+      condition: and(succeeded(), ne('${{parameters.TestsConventionOptions}}',''))
       env:
         PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat)
 
@@ -132,6 +151,6 @@ steps:
         --no-schedule
         ${{parameters.TestsConventionOptions}}
       displayName: Create Live Test pipelines for private repository
-      condition: ne('${{parameters.TestsConventionOptions}}','')
+      condition: and(succeeded(), ne('${{parameters.TestsConventionOptions}}',''))
       env:
         PATVAR: $(azuresdk-azure-sdk-devops-pipeline-generation-pat)

eng/common/scripts/stress-testing/stress-test-deployment-lib.ps1

@@ -156,9 +156,10 @@ function DeployStressPackage(
     }
     $imageTag += "/$($pkg.Namespace)/$($pkg.ReleaseName):${deployId}"
 
-    if ($pushImages) {
+    $dockerFilePath = "$($pkg.Directory)/Dockerfile"
+    if ($pushImages -and (Test-Path $dockerFilePath)) {
         Write-Host "Building and pushing stress test docker image '$imageTag'"
-        $dockerFile = Get-ChildItem "$($pkg.Directory)/Dockerfile"
+        $dockerFile = Get-ChildItem $dockerFilePath
         Run docker build -t $imageTag -f $dockerFile.FullName $dockerFile.DirectoryName
         if ($LASTEXITCODE) { return }
         Run docker push $imageTag

eng/pipelines/templates/stages/platform-matrix-cryptography-dependency.json

@@ -6,24 +6,10 @@
     "Agent": {
       "ubuntu-20.04": { "OSVmImage": "MMSUbuntu20.04", "Pool": "azsdk-pool-mms-ubuntu-2004-general" }
     },
-    "PythonVersion": [ "pypy3", "3.6", "3.7", "3.8", "3.9" ],
+    "PythonVersion": [ "pypy3", "3.7", "3.8", "3.9" ],
     "CoverageArg": "--disablecov"
   },
   "include": [
-    {
-      "Agent": {
-        "macOS-10.15": { "OSVmImage": "macOS-10.15", "Pool": "Azure Pipelines" }
-      },
-      "PythonVersion": "2.7",
-      "CoverageArg": "--disablecov"
-    },
-    {
-      "Agent": {
-        "windows-2019": { "OSVmImage": "MMS2019", "Pool": "azsdk-pool-mms-win-2019-general" }
-      },
-      "PythonVersion": "3.6",
-      "CoverageArg": "--disablecov"
-    },
     {
       "Config": {
         "Ubuntu2004_310": {

eng/pipelines/templates/stages/platform-matrix.json

@@ -10,7 +10,7 @@
       "windows-2019": { "OSVmImage": "MMS2019", "Pool": "azsdk-pool-mms-win-2019-general" },
       "macOS-10.15": { "OSVmImage": "macOS-10.15", "Pool": "Azure Pipelines" }
     },
-    "PythonVersion": [ "pypy3", "2.7", "3.6", "3.7", "3.8" ],
+    "PythonVersion": [ "pypy3", "3.7", "3.8" ],
     "CoverageArg": "--disablecov",
     "TestSamples": "false"
   },

eng/pipelines/templates/steps/analyze.yml

@@ -126,9 +126,10 @@ steps:
     parameters:
       Artifacts: ${{ parameters.Artifacts }}
 
-  - template: /eng/common/pipelines/templates/steps/detect-api-changes.yml
-    parameters:
-      Artifacts: ${{ parameters.Artifacts }}
+  # Temporarily disabling due to #22416
+  # - template: /eng/common/pipelines/templates/steps/detect-api-changes.yml
+  #   parameters:
+  #     Artifacts: ${{ parameters.Artifacts }}
 
   - template: /eng/common/pipelines/templates/steps/eng-common-workflow-enforcer.yml
 

eng/scripts/Language-Settings.ps1

@@ -204,24 +204,27 @@ function DockerValidation{
     [string]$workingDirectory
   ) 
   if ($PackageSourceOverride) {
-    docker run -v "${workingDirectory}:/workdir/out" -e TARGET_PACKAGE=$packageName -e TARGET_VERSION=$packageVersion `
-       -e EXTRA_INDEX_URL=$PackageSourceOverride -t $DocValidationImageId 2>&1 | Out-Null
+    Write-Host "docker run -v ${workingDirectory}:/workdir/out -e TARGET_PACKAGE=$packageName -e TARGET_VERSION=$packageVersion -e EXTRA_INDEX_URL=$PackageSourceOverride -t $DocValidationImageId"
+    $commandLine = docker run -v "${workingDirectory}:/workdir/out" -e TARGET_PACKAGE=$packageName -e TARGET_VERSION=$packageVersion `
+       -e EXTRA_INDEX_URL=$PackageSourceOverride -t $DocValidationImageId 2>&1 
   }
   else {
-    docker run -v "${workingDirectory}:/workdir/out" `
-      -e TARGET_PACKAGE=$packageName -e TARGET_VERSION=$packageVersion -t $DocValidationImageId 2>&1 | Out-Null
+    Write-Host "docker run -v ${workingDirectory}:/workdir/out -e TARGET_PACKAGE=$packageName -e TARGET_VERSION=$packageVersion -t $DocValidationImageId"
+    $commandLine = docker run -v "${workingDirectory}:/workdir/out" `
+      -e TARGET_PACKAGE=$packageName -e TARGET_VERSION=$packageVersion -t $DocValidationImageId 2>&1
   }
   # The docker exit codes: https://docs.docker.com/engine/reference/run/#exit-status
   # If the docker failed because of docker itself instead of the application, 
   # we should skip the validation and keep the packages. 
+
   if ($LASTEXITCODE -eq 125 -Or $LASTEXITCODE -eq 126 -Or $LASTEXITCODE -eq 127) { 
-    Write-Host $commandLine
+    $commandLine | ForEach-Object { Write-Debug $_ }
     LogWarning "The `docker` command does not work with exit code $LASTEXITCODE. Fall back to npm install $packageName directly."
     FallbackValidation -packageName "$packageName" -packageVersion "$packageVersion" -workingDirectory $workingDirectory -PackageSourceOverride $PackageSourceOverride
   }
   elseif ($LASTEXITCODE -ne 0) { 
-    Write-Host $commandLine
-    LogWarning "Package $($Package.name) ref docs validation failed."
+    $commandLine | ForEach-Object { Write-Debug $_ }
+    LogWarning "Package $packageName ref docs validation failed."
     return $false
   }
   return $true
@@ -566,4 +569,4 @@ function Validate-Python-DocMsPackages ($PackageInfo, $PackageSourceOverride, $D
   $packageVersion = $packageInfo.Version
   ValidatePackage -packageName $packageName -packageVersion $packageVersion `
       -PackageSourceOverride $PackageSourceOverride -DocValidationImageId $DocValidationImageId
-}
+}

scripts/devops_tasks/test_run_samples.py

@@ -130,6 +130,8 @@
     "azure-ai-formrecognizer": [
         "sample_manage_custom_models.py",
         "sample_manage_custom_models_async.py",
+        "sample_copy_model.py",
+        "sample_copy_model_async.py",
     ],
     "azure-ai-language-questionanswering": ["sample_chat.py"],
 }

scripts/release_issue_status/main.py

@@ -17,8 +17,9 @@
 _FILE_OUT = 'release_issue_status.csv'
 _FILE_OUT_PYTHON = 'release_python_status.md'
 _PYTHON_SDK_ADMINISTRATORS = ['msyyc', 'BigCat20196']
-_PYTHON_SDK_ASSIGNEES = ['BigCat20196']
-_ASSIGNER_DICT = {'BigCat20196': os.getenv('JF_TOKEN')}
+_PYTHON_SDK_ASSIGNEES = ['BigCat20196', 'msyyc']
+_ASSIGNER_DICT = {'BigCat20196': os.getenv('JF_TOKEN'),
+                  'msyyc': os.getenv('TOKEN')}
 logging.basicConfig(level=logging.INFO,
                     format='[auto-reply  log] - %(funcName)s[line:%(lineno)d] - %(levelname)s: %(message)s')
 

scripts/release_issue_status/release_issue_status.yml

@@ -20,7 +20,7 @@ jobs:
   displayName: ReleaseIssueStatus Python 3.8
   timeoutInMinutes: 30
   strategy:
-    maxParallel: 3
+    maxParallel: 1
   pool:
     vmImage: 'ubuntu-20.04'
   steps:
@@ -47,8 +47,6 @@ jobs:
         export URL=$(PIPELINE_URL)
         export PIPELINE_TOKEN=$(PIPELINE-TOKEN)
         export JF_TOKEN=$(Jiefeng-GitToken)
-        export ZED_TOKEN=$(Zed-GitToken)
-        export COOKIE=$(USR_TOKEN)
 
         # create virtual env
         python -m venv venv-sdk

sdk/appconfiguration/azure-appconfiguration/setup.py

@@ -59,7 +59,7 @@
     classifiers=[
         "Development Status :: 5 - Production/Stable",
         "Programming Language :: Python",
-        "Programming Language :: Python :: 3",
+        "Programming Language :: Python :: 3 :: Only",
         "Programming Language :: Python :: 3.7",
         "Programming Language :: Python :: 3.8",
         "Programming Language :: Python :: 3.9",