Refresh release/v3 branch to latest eng and scripts folder (Azure#12874)

* Refresh release/v3 branch to latest eng and scripts folder
AzureSDKAutomation · Aug 10, 2020 · 284f0c1 · 284f0c1
1 parent 11a57d5
commit 284f0c1
Show file tree

Hide file tree

Showing 344 changed files with 2,749 additions and 2,181 deletions.
diff --git a/README.md b/README.md
@@ -40,7 +40,7 @@ Libraries which enable you to provision specific resources. They are responsible
 ## Need help?
 
 * For detailed documentation visit our [Azure SDK for Python documentation](https://aka.ms/python-docs)
-* File an issue via [Github Issues](../../issues)
+* File an issue via [Github Issues](https://github.com/Azure/azure-sdk-for-python/issues)
 * Check [previous questions](https://stackoverflow.com/questions/tagged/azure+python) or ask new ones on StackOverflow using `azure` and `python` tags.
 
 ### Community

diff --git a/doc/dev/mgmt/generating-integration-test.md b/doc/dev/mgmt/generating-integration-test.md
@@ -127,7 +127,7 @@ now you can run live integration test:
 
 ## Fixing Test
 
-It's obvious that when running test for the first time someting is not going to work.
+It's obvious that when running test for the first time something is not going to work.
 
 The best approach is to:
 - fix the test manually

diff --git a/doc/dev/mgmt/generation.md b/doc/dev/mgmt/generation.md
@@ -8,7 +8,7 @@ IMPORTANT NOTE: All the commands prefixed by `python` in this page assumes you h
 
 ### Autorest versioning
 
-A few notes on [Autorest for Python versionning](https://github.com/Azure/autorest.python/blob/master/ChangeLog.md):
+A few notes on [Autorest for Python versioning](https://github.com/Azure/autorest.python/blob/master/ChangeLog.md):
 - Autorest for Python v2.x is deprecated, and should not be used anymore for any generation under any circumstances.
 - Autorest for Python v3.x is the most currently used one. Should not be used, but still ok if service team are still in v3.x and they want to avoid breaking changes for a given version (rare).
 - Autorest for Python v4.x is the current recommendation. This generator can generates async code, but this should be disabled with --no-async. No package should be shipped with async based on v4

diff --git a/doc/dev/mgmt/swagger/multi_api/readme.md b/doc/dev/mgmt/swagger/multi_api/readme.md
@@ -8,7 +8,7 @@ This is the AutoRest configuration file for Network.
 
 ## Getting Started
 
-To build the SDK for Network, simply [Install AutoRest](https://aka.ms/autorest/install) and in this folder, run:
+To build the SDK for Network, simply [Install AutoRest](https://github.com/Azure/autorest/blob/master/docs/installing-autorest.md) and in this folder, run:
 
 > `autorest`
 
@@ -66,4 +66,4 @@ swagger-to-sdk:
 
 ## Python
 
-See configuration in [readme.python.md](./readme.python.md)
+See configuration in [readme.python.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.python.md)
diff --git a/doc/dev/mgmt/swagger/single_api/readme.md b/doc/dev/mgmt/swagger/single_api/readme.md
@@ -8,7 +8,7 @@ This is the AutoRest configuration file for Cdn.
 
 ---
 ## Getting Started
-To build the SDK for Cdn, simply [Install AutoRest](https://aka.ms/autorest/install) and in this folder, run:
+To build the SDK for Cdn, simply [Install AutoRest](https://github.com/Azure/autorest/blob/master/docs/installing-autorest.md) and in this folder, run:
 
 > `autorest`
 

diff --git a/doc/dev/mgmt/swagger_conf.md b/doc/dev/mgmt/swagger_conf.md
@@ -14,42 +14,42 @@ In practical terms, we want to control the version of Autorest used, the output
 
 ## Writing the readme
 
-Writing the readme is the responsability of the Python SDK team. There is currently two types of templates for Python readmes:
+Writing the readme is the responsibility of the Python SDK team. There is currently two types of templates for Python readmes:
 - Readme that handles only one API version, and generates packages that handle one API version only
 - Readme that handles several API versions, and generates packages with multiples API and profile supports
 
-These templates can be found in the [single_api](./single_api) and the [multi_api](./multi_api) folders.
+These templates can be found in the [single_api](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/single_api) and the [multi_api](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api) folders.
 
 
 ### Single API readmes
 
 This one is the most simple:
-- Copy the [readme.python.md](./single_api/readme.python.md) and replace `servicetoreplace` by your service name
-- Be sure the main [readme.md](./single_api/readme.md) contains a "swagger-to-sdk" section with Python
+- Copy the [readme.python.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/single_api/readme.python.md) and replace `servicetoreplace` by your service name
+- Be sure the main [readme.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/single_api/readme.md) contains a "swagger-to-sdk" section with Python
 
 ### Multi API readmes
 
-When doing multi-api packages, it means you have shipping several "tags" of the main readme as one package. Autorest is calling this process a "batch" call, and this is the purpose of the "batch" section in [readme.python.md](./multi_api/readme.python.md).
+When doing multi-api packages, it means you have shipping several "tags" of the main readme as one package. Autorest is calling this process a "batch" call, and this is the purpose of the "batch" section in [readme.python.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.python.md).
 
 In order to be sure the correct tags exist, you can use the following script:
 ```shell
 python ./scripts/multi_api_readme_help.py /azure-rest-api-specs/specification/service/resource-manager/
 ```
 
 This script will analyze the Swaggers available, and suggests on stdout:
-- A list of tags for the main [readme.md](./multi_api/readme.md)
-- A batch declaration for the [readme.python.md](./multi_api/readme.python.md)
+- A list of tags for the main [readme.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.md)
+- A batch declaration for the [readme.python.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.python.md)
 
 This script is not perfect that it *does* require manual review of the output and not a direct copy/paste.
 
 It's important for Python that tags represents only *ONE* unique API version. It's why it's pretty common that Python uses a set of tags that other languages don't use.
 
 Once you know the list of tags you need to generate:
 
-- Copy the [readme.python.md](./multi_api/readme.python.md) and replace `servicetoreplace` by your service name
-- Update the batch list of [readme.python.md](./multi_api/readme.python.md)
-- Be sure you have one tag section for each batch entry in [readme.python.md](./multi_api/readme.python.md)
-- Be sure the main [readme.md](./multi_api/readme.md) contains a "swagger-to-sdk" section with Python with an `afterscripts` section like the one in the template.
+- Copy the [readme.python.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.python.md) and replace `servicetoreplace` by your service name
+- Update the batch list of [readme.python.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.python.md)
+- Be sure you have one tag section for each batch entry in [readme.python.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.python.md)
+- Be sure the main [readme.md](https://github.com/Azure/azure-sdk-for-python/blob/master/doc/dev/mgmt/swagger/multi_api/readme.md) contains a "swagger-to-sdk" section with Python with an `afterscripts` section like the one in the template.
 
 The `afterscripts` will execute a Jinja template to create a client to link together all the batch generated autorest ([example](https://github.com/Azure/azure-sdk-for-python/blob/4a7c67189591b052fe2b5769847ff68f7845386d/sdk/storage/azure-mgmt-storage/azure/mgmt/storage/_storage_management_client.py))
 

diff --git a/eng/.docsettings.yml b/eng/.docsettings.yml
@@ -43,6 +43,10 @@ known_content_issues:
   # data plane. common issue filed to azure-sdk-for-python
   - ['README.rst', '#4554']
   - ['sdk/synapse/azure-synapse/README.md', '#4554']
+  - ['sdk/synapse/azure-synapse-accesscontrol/README.md', '#4554']
+  - ['sdk/synapse/azure-synapse-spark/README.md', '#4554']
+  - ['sdk/synapse/azure-synapse-artifacts/README.md', '#4554']
+  - ['sdk/synapse/azure-synapse-nspkg/README.md', '#4554']
   - ['sdk/applicationinsights/azure-applicationinsights/README.md', '#4554']
   - ['sdk/batch/azure-batch/README.md', '#4554']
   - ['sdk/cognitiveservices/azure-cognitiveservices-anomalydetector/README.md', '#4554']
@@ -78,6 +82,7 @@ known_content_issues:
   - ['sdk/graphrbac/azure-graphrbac/README.md', '#4554']
   - ['sdk/loganalytics/azure-loganalytics/README.md', '#4554']
   - ['sdk/servicebus/azure-servicebus/README.md', '#4554']
+  - ['sdk/servicebus/azure-servicebus/swagger/README.md', '#4554']
   - ['sdk/servicefabric/azure-servicefabric/README.md', '#4554']
   - ['sdk/storage/azure-storage-nspkg/README.md', '#4554']
   - ['sdk/storage/azure-storage-blob/swagger/README.md', '#4554']
@@ -109,10 +114,12 @@ known_content_issues:
   - ['sdk/search/azure-search-nspkg/README.md', 'nspkg and common']
   - ['sdk/storage/azure-storage-blob/samples/README.md', 'nspkg and common']
   - ['sdk/storage/azure-storage-file-datalake/samples/README.md', 'nspkg and common']
+  - ['sdk/storage/azure-storage-blob-changefeed/samples/README.md', 'nspkg and common']
   - ['sdk/storage/azure-storage-file-share/samples/README.md', 'nspkg and common']
   - ['sdk/storage/azure-storage-queue/samples/README.md', 'nspkg and common']
   - ['sdk/textanalytics/azure-ai-nspkg/README.md', 'nspkg and common']
   - ['sdk/eventhub/azure-eventhub/samples/README.md', 'nspkg and common']
+  - ['sdk/tables/azure-data-nspkg/README.md', 'nspkg and common']
 
 # HISTORY.rst
   - ['sdk/core/azure/HISTORY.rst','#4554']

diff --git a/eng/ci_tools.txt b/eng/ci_tools.txt
@@ -1,36 +1,35 @@
 # requirements leveraged by ci tools
 setuptools==44.1.0; python_version == '2.7'
-setuptools==45.1.0; python_version >= '3.5'
+setuptools==46.4.0; python_version >= '3.5'
 virtualenv==20.0.23
-wheel==0.34.2
-Jinja2==2.11.1
+wheel==0.34.2 
+Jinja2==2.11.2
 packaging==20.4
-tox==3.14.6
+tox==3.15.0
 tox-monorepo==0.1.2
-twine==1.15.0
+twine==1.15.0; python_version == '2.7' or python_version == '3.5'
+twine==3.1.1; python_version >= '3.6'
 pathlib2==2.3.5
 readme-renderer[md]==25.0
-doc-warden==0.5.4
+doc-warden==0.7.1
+# we pin coverage to 4.5.4 because there is an bug with `pytest-cov`. the generated coverage files cannot be `coverage combine`ed
 coverage==4.5.4
-codecov==2.0.22 
-beautifulsoup4==4.8.2
+codecov==2.1.0
+beautifulsoup4==4.9.1
 pkginfo==1.5.0.1
 
 # locking packages defined as deps from azure-sdk-tools or azure-devtools
 pytoml==0.1.21
 pyOpenSSL==19.1.0
 json-delta==2.0
-ConfigArgParse==1.1
+ConfigArgParse==1.2.3
 six==1.14.0
 vcrpy==3.0.0
 pyyaml==5.3.1
-pytest==5.4.1; python_version >= '3.5'
+pytest==5.4.2; python_version >= '3.5'
 pytest==4.6.9; python_version == '2.7'
 pytest-cov==2.8.1
 
 # local dev packages
 ./tools/azure-devtools
 ./tools/azure-sdk-tools
-
-
-
diff --git a/eng/common/Extract-ReleaseNotes.ps1 b/eng/common/Extract-ReleaseNotes.ps1
diff --git a/eng/common/TestResources/New-TestResources.ps1 b/eng/common/TestResources/New-TestResources.ps1
@@ -154,14 +154,21 @@ if ($ProvisionerApplicationId) {
     $subscriptionArgs = if ($SubscriptionId) {
         @{SubscriptionId = $SubscriptionId}
     }
+    else {
+        @{}
+    }
 
     $provisionerAccount = Retry {
         Connect-AzAccount -Force:$Force -Tenant $TenantId -Credential $provisionerCredential -ServicePrincipal -Environment $Environment @subscriptionArgs
     }
 
     $exitActions += {
         Write-Verbose "Logging out of service principal '$($provisionerAccount.Context.Account)'"
-        $null = Disconnect-AzAccount -AzureContext $provisionerAccount.Context
+
+        # Only attempt to disconnect if the -WhatIf flag was not set.  Otherwise, this call is not necessary and will fail.
+        if ($PSCmdlet.ShouldProcess($ProvisionerApplicationId)) {
+            $null = Disconnect-AzAccount -AzureContext $provisionerAccount.Context
+        }
     }
 }
 
@@ -176,7 +183,6 @@ if ($TestApplicationId -and !$TestApplicationOid) {
     }
 }
 
-
 # If the ServiceDirectory is an absolute path use the last directory name
 # (e.g. D:\foo\bar\ -> bar)
 $serviceName = if (Split-Path -IsAbsolute  $ServiceDirectory) {
@@ -229,6 +235,13 @@ if ($resourceGroup.ProvisioningState -eq 'Succeeded') {
     # New-AzResourceGroup would've written an error and stopped the pipeline by default anyway.
     Write-Verbose "Successfully created resource group '$($resourceGroup.ResourceGroupName)'"
 }
+elseif (($resourceGroup -eq $null) -and (-not $PSCmdlet.ShouldProcess($resourceGroupName))) {
+    # If the -WhatIf flag was passed, there will be no resource group created.  Fake it.
+    $resourceGroup = [PSCustomObject]@{
+        ResourceGroupName = $resourceGroupName
+        Location = $Location
+    }
+}
 
 # Populate the template parameters and merge any additional specified.
 $templateParameters = @{
@@ -299,6 +312,9 @@ foreach ($templateFile in $templateFiles) {
         "$($serviceDirectoryPrefix)RESOURCE_GROUP" = $resourceGroup.ResourceGroupName;
         "$($serviceDirectoryPrefix)LOCATION" = $resourceGroup.Location;
         "$($serviceDirectoryPrefix)ENVIRONMENT" = $context.Environment.Name;
+        "$($serviceDirectoryPrefix)AZURE_AUTHORITY_HOST" = $context.Environment.ActiveDirectoryAuthority;
+        "$($serviceDirectoryPrefix)RESOURCE_MANAGER_URL" = $context.Environment.ResourceManagerUrl;
+        "$($serviceDirectoryPrefix)SERVICE_MANAGEMENT_URL" = $context.Environment.ServiceManagementUrl;
     }
 
     foreach ($key in $deployment.Outputs.Keys) {
@@ -331,7 +347,7 @@ foreach ($templateFile in $templateFiles) {
     }
     else
     {
-        
+
         if (!$CI) {
             # Write an extra new line to isolate the environment variables for easy reading.
             Log "Persist the following environment variables based on your detected shell ($shell):`n"
@@ -340,7 +356,7 @@ foreach ($templateFile in $templateFiles) {
         foreach ($key in $deploymentOutputs.Keys)
         {
             $value = $deploymentOutputs[$key]
-            
+
             if ($CI) {
                 # Treat all ARM template output variables as secrets since "SecureString" variables do not set values.
                 # In order to mask secrets but set environment variables for any given ARM template, we set variables twice as shown below.

diff --git a/eng/common/TestResources/New-TestResources.ps1.md b/eng/common/TestResources/New-TestResources.ps1.md
@@ -412,6 +412,23 @@ Accept pipeline input: False
 Accept wildcard characters: False
 ```
 
+### -OutFile
+save test environment settings into a test-resources.json.env file next to test-resources.json. 
+The file is protected via DPAPI. The environment file would be scoped to the current repository directory.
+Note: Supported only on Windows.
+
+```yaml
+Type: SwitchParameter
+Parameter Sets: (All)
+Aliases:
+
+Required: False
+Position: Named
+Default value: None
+Accept pipeline input: False
+Accept wildcard characters: False
+```
+
 ### CommonParameters
 This cmdlet supports the common parameters: -Debug, -ErrorAction, -ErrorVariable, -InformationAction, -InformationVariable, -OutVariable, -OutBuffer, -PipelineVariable, -Verbose, -WarningAction, and -WarningVariable. For more information, see [about_CommonParameters](http://go.microsoft.com/fwlink/?LinkID=113216).
 

diff --git a/eng/common/TestResources/README.md b/eng/common/TestResources/README.md
@@ -35,6 +35,12 @@ eng\common\TestResources\New-TestResources.ps1 `
   -TestApplicationSecret (ConvertFrom-SecureString $sp.Secret -AsPlainText)
 ```
 
+If you are running this for a .NET project on Windows, the recommended method is to 
+add the `-OutFile` switch to the above command. This will save test environment settings 
+into a test-resources.json.env file next to test-resources.json. The file is protected via DPAPI. 
+The environment file would be scoped to the current repository directory and avoids the need to 
+set environment variables or restart your IDE to recognize them.
+
 Along with some log messages, this will output environment variables based on
 your current shell like in the following example:
 

diff --git a/eng/common/TestResources/deploy-test-resources.yml b/eng/common/TestResources/deploy-test-resources.yml
@@ -13,7 +13,7 @@ parameters:
 #   "TestApplicationId": "<test app id>",
 #   "TestApplicationSecret": "<test app secret>",
 #   "ProvisionerApplicationId": "<provisoner app id>",
-#   "ProvisoinerApplicationSecret": "<provisoner app secert>",
+#   "ProvisionerApplicationSecret": "<provisoner app secert>",
 #   "Environment": "AzureCloud | AzureGov | AzureChina | <other environment>"
 # }
 

diff --git a/eng/common/TestResources/remove-test-resources.yml b/eng/common/TestResources/remove-test-resources.yml
@@ -32,5 +32,5 @@ steps:
         -Force `
         -Verbose
     displayName: Remove test resources
-    condition: and(ne(variables['AZURE_RESOURCEGROUP_NAME'], ''), succeededOrFailed())
+    condition: ne(variables['AZURE_RESOURCEGROUP_NAME'], '')
     continueOnError: true