Merge branch 'main' of https://github.com/Azure/azure-sdk-for-python …

…into fix_awaiting

* 'main' of https://github.com/Azure/azure-sdk-for-python: (57 commits)
  Add latest released version to prepare prerelease (Azure#21420)
  [textanalytics] feature/ta-v3.2-preview.2 (Azure#21409)
  Update reply comment (Azure#21399)
  [AutoRelease] t2-cosmosdb-2021-10-26-16894 (Azure#21412)
  [AutoRelease] t2-containerregistry-2021-10-26-15283 (Azure#21410)
  [AutoRelease] t2-azurearcdata-2021-10-14-24172 (Azure#21251)
  [AVA] Updated sdk to 1.1 (Azure#21326)
  [AutoRelease] t2-chaos-2021-10-25-61789 (Azure#21390)
  [AutoRelease] t2-keyvault-2021-10-15-69767 (Azure#21276)
  [formrecognizer] Rename DocumentElement to DocumentContentElement (Azure#21323)
  update error code in tests (Azure#21362)
  Fully automate cluster buildout. Add azure file share mount to stress tests. (Azure#21382)
  [Cosmos] get_user_client() method documentation update (Azure#21379)
  Removed auto pagination (Azure#21358)
  [AutoRelease] t2-network-2021-10-21-48085 (Azure#21357)
  patching last references to master in the repo! (Azure#21339)
  Update Language-Settings.ps1 (Azure#21367)
  Remove azure-mgmt-webpubsub (Azure#21347)
  [SchemaRegistry] add async version of avro serializer (Azure#21026)
  move async test to _async (Azure#21363)
  ...

.github/CODEOWNERS

@@ -41,6 +41,7 @@
 /sdk/communication/                                                  @acsdevx-msft
 /sdk/communication/azure-communication-phonenumbers/                 @RoyHerrod @danielav7 @whisper6284 @AlonsoMondal
 /sdk/communication/azure-communication-sms/                          @RoyHerrod @arifibrahim4
+/sdk/communication/azure-communication-identity/                     @Azure/acs-identity-sdk
 
 # PRLabel: %KeyVault
 /sdk/keyvault/                                                       @schaabs @chlowell @mccoyp @YalinLi0312
@@ -65,7 +66,7 @@
 /sdk/containerservice/                                               @samkreter @zqingqing1 @GaneshaThirumurthi
 
 # PRLabel: %Cosmos
-/sdk/cosmos/                                                         @southpolesteve @zfoster
+/sdk/cosmos/                                                         @kushagraThapar @simorenoh @simplynaveen20 @xinlian12 @moderakh 
 
 # PRLabel: %Data Factory
 /sdk/datafactory/                                                    @hvermis

doc/dev/README.md

@@ -7,7 +7,7 @@ Overview of the documents:
 - [Release](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/release.md) : How to release a package when ready
 - [Packaging](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/packaging.md) : How to organize packaging information for packages under `azure`
 - [Testing](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/tests.md): How to write unit and functional tests for a library
-- [Docstrings and Type hints](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/docstring_typehint.md): How to document an SDK for compatability with internal tools (API View) and our documentation at [MS Docs][ms_docs] and the [azure.github.io][azure_github_io] site.
+- [Docstrings and Type hints](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/docstring_typehint.md): How to document an SDK for compatibility with internal tools (API View) and our documentation at [MS Docs][ms_docs] and the [azure.github.io][azure_github_io] site.
 
 The [mgmt](https://github.com/Azure/azure-sdk-for-python/blob/main/doc/dev/mgmt) folder contains information specific to management packages (i.e. packages prefixed by `azure-mgmt`)
 

doc/dev/get_unreleased_package_guide.md

@@ -16,4 +16,4 @@ The following figure shows the wheel and zip of the package.Click to download th
 (1) If there is no link in the figure above, it may be folded. You can also find it in the check.
 ![img.png](unreleased_package_guide_example3.png)
 
-(2) [Private repo](https://github.com/Azure/azure-rest-api-specs-pr) can only be triggered when the target branch is `main`
+(2) The private Azure/azure-rest-api-specs-pr repo can only be triggered when the target branch is `main`

doc/dev/perfstress_tests.md

@@ -82,7 +82,7 @@ The framework has a series of common command line options built in:
 - `-w --warm-up=5` Number of seconds to spend warming up the connection before measuring begins. Default is 5.
 - `--sync` Whether to run the tests in sync or async. Default is False (async).
 - `--no-cleanup` Whether to keep newly created resources after test run. Default is False (resources will be deleted).
-- `-x --test-proxy` Whether to run the tests against the test proxy server. Specfiy the URL for the proxy endpoint (e.g. "https://localhost:5001").
+- `-x --test-proxies` Whether to run the tests against the test proxy server. Specify the URL(s) for the proxy endpoint(s) (e.g. "https://localhost:5001").
 - `--profile` Whether to run the perftest with cProfile. If enabled (default is False), the output file of the **last completed single iteration** will be written to the current working directory in the format `"cProfile-<TestClassName>-<TestID>-<sync/async>.pstats"`.
 
 

eng/common/pipelines/templates/steps/get-pr-owners.yml

@@ -1,27 +1,41 @@
 parameters: 
   TargetVariable: ''
+  TargetUserVariable: 'notspecified'
+  TargetTeamVariable: 'notspecified'
+  TargetLabelVariable: 'notspecified'
   ServiceDirectory: ''
+  DevOpsFeed: "https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-net/nuget/v3/index.json"
 
 steps:
-  - pwsh: |
-      git clone https://github.com/Azure/azure-sdk-tools.git $(Build.SourcesDirectory)/tools_repo
-      cd $(Build.SourcesDirectory)/tools_repo
-      git checkout azure-sdk-tools_20210114.1
-    displayName: Setup Identity Resolver
+  - task: DotNetCoreCLI@2
+    displayName: 'Install Identity Resolver'
+    inputs:
+      command: custom
+      custom: 'tool'
+      arguments: 'install --global --add-source "${{ parameters.DevOpsFeed }}" --version "1.0.0-dev.20211018.1" "Azure.Sdk.Tools.IdentityResolver"'
+      workingDirectory: '$(Agent.BuildDirectory)'
+
+  - task: DotNetCoreCLI@2
+    displayName: 'Install CodeOwners Retriever'
+    inputs:
+      command: custom
+      custom: 'tool'
+      arguments: 'install --global --add-source "${{ parameters.DevOpsFeed }}" --version "1.0.0-dev.20211019.1" "Azure.Sdk.Tools.RetrieveCodeOwners"'
+      workingDirectory: '$(Agent.BuildDirectory)'
 
   - pwsh: |
-      dotnet run -v q -- `
+      identity-resolver `
         --aad-app-id-var APP_ID `
         --aad-app-secret-var APP_SECRET `
         --aad-tenant-var AAD_TENANT `
         --kusto-url-var KUSTO_URL `
         --kusto-database-var KUSTO_DB `
         --kusto-table-var KUSTO_TABLE `
-        --identity "$(Build.QueuedBy)" `
-        --targetvar "${{ parameters.TargetVariable }}"
+        --identity-name "$(Build.QueuedBy)" `
+        --identity-email "$(Build.RequestedForEmail)" `
+        --targetvar "${{ coalesce(parameters.TargetVariable, parameters.TargetUserVariable) }}"
     displayName: 'Resolving Queuing User'
     continueOnError: true
-    workingDirectory: $(Build.SourcesDirectory)/tools_repo/tools/notification-configuration/identity-resolver
     env:
       APP_ID: $(notification-aad-app-id)
       APP_SECRET: $(notification-aad-secret)
@@ -31,15 +45,10 @@ steps:
       KUSTO_TABLE: $(notification-kusto-table)
 
   - pwsh: |
-      Remove-Item -Force -Recurse $(Build.SourcesDirectory)/tools_repo
-    displayName: Clean Up Cloned Tools Repo
-
-  - task: PowerShell@2
-    displayName: Add CodeOwners if Present
-    inputs:
-      pwsh: true
-      filePath: $(Build.SourcesDirectory)/eng/common/scripts/get-codeowners.ps1
-      arguments: >
-        -TargetDirectory "/sdk/${{ parameters.ServiceDirectory }}/"
-        -RootDirectory "$(Build.SourcesDirectory)"
-        -VsoVariable "${{ parameters.TargetVariable }}"
+      retrieve-codeowners `
+        --target-directory "/sdk/${{ parameters.ServiceDirectory }}/" `
+        --root-directory "$(Build.SourcesDirectory)" `
+        --vso-owning-users "${{ coalesce(parameters.TargetVariable, parameters.TargetUserVariable) }}" `
+        --vso-owning-teams "${{ parameters.TargetTeamVariable }}" `
+        --vso-owning-labels "${{ parameters.TargetLabelVariable }}"
+    displayName: 'Add CodeOwners if Present'

eng/common/scripts/Prepare-Release.ps1

@@ -109,11 +109,15 @@ else
 $releaseDateString = $ParsedReleaseDate.ToString("MM/dd/yyyy")
 $month = $ParsedReleaseDate.ToString("MMMM")
 
-Write-Host
 Write-Host "Assuming release is in $month with release date $releaseDateString" -ForegroundColor Green
+if (Test-Path "Function:GetExistingPackageVersions")
+{
+    $releasedVersions = GetExistingPackageVersions -PackageName $packageProperties.Name -GroupId $packageProperties.Group
+    $latestReleasedVersion = $releasedVersions[$releasedVersions.Count - 1]
+    Write-Host "Latest released version: ${latestReleasedVersion}" -ForegroundColor Green
+}
 
 $currentProjectVersion = $packageProperties.Version
-
 $newVersion = Read-Host -Prompt "Input the new version, or press Enter to use use current project version '$currentProjectVersion'"
 
 if (!$newVersion)

eng/common/scripts/Update-DocsMsPackages.ps1

@@ -54,14 +54,19 @@ function GetDocsMetadataForMoniker($moniker) {
       $versionPreview = $fileObject.Version
     }
 
-    $metadata += @{ 
+    $entry = @{
       Package = $fileObject.Name; 
       VersionGA = $versionGa;
       VersionPreview = $versionPreview;
       RepoPath = $fileObject.ServiceDirectory;
       Type = $fileObject.SdkType;
       New = $fileObject.IsNewSdk;
     }
+    if ($fileObject.PSObject.Members.Name -contains "Group")
+    {
+      $entry.Add("GroupId", $fileObject.Group)
+    }
+    $metadata += $entry
   }
 
   return $metadata

eng/common/scripts/stress-testing/deploy-stress-tests.ps1

@@ -76,9 +76,9 @@ function DeployStressTests(
     [string]$environment = 'test',
     [string]$repository = 'images',
     [boolean]$pushImages = $false,
-    [string]$clusterGroup = 'rg-stress-test-cluster-',
+    [string]$clusterGroup = 'rg-stress-cluster-test',
     [string]$deployId = 'local',
-    [string]$subscription = 'Azure SDK Test Resources'
+    [string]$subscription = 'Azure SDK Developer Playground'
 ) {
     if ($PSCmdlet.ParameterSetName -eq 'DoLogin') {
         Login $subscription $clusterGroup $pushImages

eng/common/testproxy/docker-start-proxy.ps1

@@ -25,7 +25,7 @@ catch {
     Write-Error "Please check your docker invocation and try running the script again."
 }
 
-$SELECTED_IMAGE_TAG = "1084681"
+$SELECTED_IMAGE_TAG = "1147815"
 $CONTAINER_NAME = "ambitious_azsdk_test_proxy"
 $LINUX_IMAGE_SOURCE = "azsdkengsys.azurecr.io/engsys/testproxy-lin:${SELECTED_IMAGE_TAG}"
 $WINDOWS_IMAGE_SOURCE = "azsdkengsys.azurecr.io/engsys/testproxy-win:${SELECTED_IMAGE_TAG}"
@@ -40,13 +40,15 @@ function Get-Proxy-Container(){
 
 $SelectedImage = $LINUX_IMAGE_SOURCE
 $Initial = ""
+$LinuxContainerArgs = "--add-host=host.docker.internal:host-gateway"
 
 # most of the time, running this script on a windows machine will work just fine, as docker defaults to linux containers
 # however, in CI, windows images default to _windows_ containers. We cannot swap them. We can tell if we're in a CI build by
 # checking for the environment variable TF_BUILD.
 if ($IsWindows -and $env:TF_BUILD){
     $SelectedImage = $WINDOWS_IMAGE_SOURCE
     $Initial = "C:"
+    $LinuxContainerArgs = ""
 }
 
 if ($Mode -eq "start"){
@@ -63,8 +65,8 @@ if ($Mode -eq "start"){
     # else we need to create it
     else {
         Write-Host "Attempting creation of Docker host $CONTAINER_NAME"
-        Write-Host "docker container create -v `"${root}:${Initial}/etc/testproxy`" -p 5001:5001 -p 5000:5000 --name $CONTAINER_NAME $SelectedImage"
-        docker container create -v "${root}:${Initial}/etc/testproxy" -p 5001:5001 -p 5000:5000 --name $CONTAINER_NAME $SelectedImage
+        Write-Host "docker container create -v `"${root}:${Initial}/etc/testproxy`" $LinuxContainerArgs -p 5001:5001 -p 5000:5000 --name $CONTAINER_NAME $SelectedImage"
+        docker container create -v "${root}:${Initial}/etc/testproxy" $LinuxContainerArgs -p 5001:5001 -p 5000:5000 --name $CONTAINER_NAME $SelectedImage
     }
 
     Write-Host "Attempting start of Docker host $CONTAINER_NAME"

eng/scripts/Language-Settings.ps1

@@ -206,7 +206,9 @@ function ValidatePackage($packageName, $packageVersion, $workingDirectory) {
   return $true
 }
 
-$PackageExclusions = @{ 
+$PackageExclusions = @{
+  'azure-mgmt-quota' = 'Unsupported doc directives: https://github.com/Azure/azure-sdk-for-python/issues/21366';
+  'azure-mgmt-webpubsub' = 'Unsupported doc directives https://github.com/Azure/azure-sdk-for-python/issues/21346';
   'azure-mgmt-apimanagement' = 'Unsupported doc directives https://github.com/Azure/azure-sdk-for-python/issues/18084';
   'azure-mgmt-reservations' = 'Unsupported doc directives https://github.com/Azure/azure-sdk-for-python/issues/18077';
   'azure-mgmt-signalr' = 'Unsupported doc directives https://github.com/Azure/azure-sdk-for-python/issues/18085';

scripts/release_issue_status/main.py

@@ -17,7 +17,9 @@
 _NULL = ' '
 _FILE_OUT = 'release_issue_status.csv'
 _FILE_OUT_PYTHON = 'release_python_status.md'
-_PYTHON_SDK_ADMINISTRATORS = {'msyyc', 'RAY-316', 'BigCat20196'}
+_PYTHON_SDK_ADMINISTRATORS = ['msyyc', 'RAY-316', 'BigCat20196']
+_PYTHON_SDK_ASSIGNEES = ['RAY-316', 'BigCat20196']
+_ASSIGNER_DICT = {'RAY-316': os.getenv('ZED_TOKEN'), 'BigCat20196': os.getenv('JF_TOKEN')}
 logging.basicConfig(level=logging.INFO,
                     format='[auto-reply  log] - %(funcName)s[line:%(lineno)d] - %(levelname)s: %(message)s')
 
@@ -128,7 +130,7 @@ def _latest_comment_time(comments, delay_from_create_date):
     return delay_from_create_date if not q else int((time.time() - q[-1][0]) / 3600 / 24)
 
 
-def auto_reply(item, request_repo, rest_repo, sdk_repo, duplicated_issue, python_piplines):
+def auto_reply(item, request_repo, rest_repo, sdk_repo, duplicated_issue, python_piplines, assigner_repoes):
     logging.info("new issue number: {}".format(item.issue_object.number))
 
     if 'auto-link' not in item.labels:
@@ -156,8 +158,9 @@ def auto_reply(item, request_repo, rest_repo, sdk_repo, duplicated_issue, python
     try:
         logging.info(python_piplines)
         pipeline_url = get_pipeline_url(python_piplines, output_folder)
+        assigner_repo = assigner_repoes[item.assignee]
         rg.begin_reply_generate(item=item, rest_repo=rest_repo, readme_link=readme_link,
-                                sdk_repo=sdk_repo, pipeline_url=pipeline_url)
+                                sdk_repo=sdk_repo, pipeline_url=pipeline_url, assigner_repo=assigner_repo)
         if 'Configured' in item.labels:
             item.issue_object.remove_from_labels('Configured')
     except Exception as e:
@@ -170,8 +173,11 @@ def auto_reply(item, request_repo, rest_repo, sdk_repo, duplicated_issue, python
 def main():
     # get latest issue status
     g = Github(os.getenv('TOKEN'))  # please fill user_token
+    assigner_repoes = {}
+    for k, v in _ASSIGNER_DICT.items():
+        assigner_repoes[k] = Github(v).get_repo('Azure/sdk-release-request')
     request_repo = g.get_repo('Azure/sdk-release-request')
-    rest_repo = g.get_repo('Azure/azure-rest-api-specs')   
+    rest_repo = g.get_repo('Azure/azure-rest-api-specs')
     sdk_repo = g.get_repo('Azure/azure-sdk-for-python')
     label1 = request_repo.get_label('ManagementPlane')
     open_issues = request_repo.get_issues(state='open', labels=[label1])
@@ -226,13 +232,13 @@ def main():
             item.bot_advice = 'new issue and better to confirm quickly.'
             if 'assigned' not in item.labels:
                 time.sleep(0.1)
-                assign_count = int(str(time.time())[-1]) % 2
-                if assign_count == 1:
-                    item.issue_object.remove_from_assignees(*['RAY-316'])
-                    item.issue_object.add_to_assignees(*['BigCat20196'])
+                assign_count = int(str(time.time())[-1]) % len(_PYTHON_SDK_ASSIGNEES)
+                item.issue_object.remove_from_assignees(item.assignee)
+                item.issue_object.add_to_assignees(_PYTHON_SDK_ASSIGNEES[assign_count])
+                item.assignee=item.issue_object.assignee.login
                 item.issue_object.add_to_labels('assigned')
             try:
-                auto_reply(item, request_repo, rest_repo, sdk_repo, duplicated_issue, python_piplines)
+                auto_reply(item, request_repo, rest_repo, sdk_repo, duplicated_issue, python_piplines, assigner_repoes)
             except Exception as e:
                 continue
         elif not item.author_latest_comment in _PYTHON_SDK_ADMINISTRATORS:

scripts/release_issue_status/release_issue_status.yml

@@ -39,6 +39,8 @@ jobs:
         export HEADERS=$(PIPELINE_HEADERS)
         export URL=$(PIPELINE_URL)
         export PIPELINE_TOKEN=$(PIPELINE-TOKEN)
+        export JF_TOKEN=$(Jiefeng-GitToken)
+        export ZED_TOKEN=$(Zed-GitToken)
         export COOKIE=$(USR_TOKEN)
 
         # create virtual env

scripts/release_issue_status/reply_generator.py

@@ -52,8 +52,10 @@ def get_latest_pr_from_readme(rest_repo, link_dict):
     return latest_pr_number_int[-1]
 
 
-def reply_owner(reply_content):
-    issue_object_rg.create_comment(reply_content)
+def reply_owner(assigner_issue, reply_content):
+    assigner_issue.create_comment(reply_content)
+    user_tips = 'Tips: If you have special needs for release date or other things, please let us know. Otherwise we will release it ASAP after your check.'
+    assigner_issue.create_comment(user_tips)
 
 
 
@@ -74,7 +76,7 @@ def get_reply_and_sdk_number_from_readme(rest_repo, link_dict):
     return info_model, sdk_link_number
 
 
-def begin_reply_generate(item, rest_repo, readme_link, sdk_repo, pipeline_url):
+def begin_reply_generate(item, rest_repo, readme_link, sdk_repo, pipeline_url, assigner_repo):
     global issue_object_rg
     issue_object_rg = item.issue_object
     link_dict = get_links(readme_link)
@@ -91,7 +93,8 @@ def begin_reply_generate(item, rest_repo, readme_link, sdk_repo, pipeline_url):
             logging.info(f'{issue_object_rg.number} run pipeline successfully')
         else:
             logging.info(f'{issue_object_rg.number} run pipeline fail')
-        reply_owner(reply_content)
+        assigner_issue = assigner_repo.get_issue(number=issue_object_rg.number)
+        reply_owner(assigner_issue, reply_content)
         issue_object_rg.add_to_labels('auto-ask-check')
     else:
         logging.info('issue {} need config readme'.format(issue_object_rg.number))

sdk/appconfiguration/azure-appconfiguration/CHANGELOG.md

@@ -1,13 +1,22 @@
 # Release History
 
-## 1.2.1 (Unreleased)
+## 1.3.0 (Unreleased)
 
 ### Features Added
 
 ### Breaking Changes
 
 ### Bugs Fixed
 
+- Fix the issue that data was persisted according to an incorrect schema/in an incorrect format ([#20518](https://github.com/Azure/azure-sdk-for-python/issues/20518))
+
+  `SecretReferenceConfigurationSetting` in 1.2.0 used "secret_uri" rather than "uri" as the schema keywords which 
+  broken inter-operation of `SecretReferenceConfigurationSetting` between SDK and the portal. 
+
+  Please:
+  - Use 1.3.0+ for any `SecretReferenceConfigurationSetting` uses.
+  - Call a get method for existing `SecretReferenceConfigurationSetting`s and set them back to correct the format.
+
 ### Other Changes
 
 ## 1.2.0 (2021-07-06)