Merge remote-tracking branch 'origin/feature/eventhubs_refactor' into…

… anna-auth

.docsettings.yml

@@ -51,6 +51,7 @@ known_content_issues:
   - ['sdk/storage/azure-storage-file-share/swagger/README.md', '#4554']
   - ['sdk/storage/azure-storage-queue/swagger/README.md', '#4554']
   - ['sdk/storage/azure-storage-blob/swagger/README.md', '#4554']
+  - ['sdk/storage/azure-storage-file-datalake/swagger/README.md', '#4554']
 
   # nspckg and common.
   - ['sdk/cognitiveservices/azure-cognitiveservices-nspkg/README.rst', 'nspkg and common']
@@ -69,6 +70,7 @@ known_content_issues:
   - ['sdk/storage/azure-storage-blob/samples/README.md', 'nspkg and common']
   - ['sdk/storage/azure-storage-queue/samples/README.md', 'nspkg and common']
   - ['sdk/storage/azure-storage-file-share/samples/README.md', 'nspkg and common']
+  - ['sdk/storage/azure-storage-file-datalake/samples/README.md', 'nspkg and common']
   - ['sdk/appconfiguration/azure-appconfiguration/README.md', 'nspkg and common']
   - ['sdk/appconfiguration/azure-appconfiguration/samples/README.md', 'nspkg and common']
   - ['sdk/keyvault/azure-keyvault-nspkg/README.md', 'nspkg and common']

.gitignore

@@ -90,5 +90,6 @@ code_reports
 sdk/storage/azure-storage-blob/tests/settings_real.py
 sdk/storage/azure-storage-queue/tests/settings_real.py
 sdk/storage/azure-storage-file-share/tests/settings_real.py
+sdk/storage/azure-storage-file-datalake/tests/settings_real.py
 *.code-workspace
 sdk/cosmos/azure-cosmos/test/test_config.py

README.md

@@ -17,30 +17,30 @@ The client libraries are supported on Python 2.7 and 3.5.3 or later.
 
 ## Packages available
 Each service might have a number of libraries available from each of the following categories:
-* [Client - July 2019 Preview](#Client-July-2019-Preview)
-* [Client - Stable](#Client-Stable)
+* [Client - November 2019 Releases](#Client-November-2019-Releases)
+* [Client - Previous Versions](#Client-Previous-Versions)
 * [Management](#Management)
 
 
-### Client: July 2019 Preview
-New wave of packages that we are currently releasing in **Preview**. These libraries allow you to use and consume existing resources and interact with them, for example: upload a blob. These libraries share a number of core functionalities such as: retries, logging, transport protocols, authentication protocols, etc. that can be found in the [azure-core](./sdk/core/azure-core) library. You can learn more about these libraries by reading guidelines that they follow [here](https://azuresdkspecs.z5.web.core.windows.net/PythonSpec.html).
+### Client: November 2019 Releases
+New wave of packages that we are announcing as **GA** and several that are currently releasing in **preview**. These libraries allow you to use and consume existing resources and interact with them, for example: upload a blob. These libraries share a number of core functionalities such as: retries, logging, transport protocols, authentication protocols, etc. that can be found in the [azure-core](./sdk/core/azure-core) library. You can learn more about these libraries by reading guidelines that they follow [here](https://azure.github.io/azure-sdk/python_introduction.html).
 
-The libraries released in July preview:
-
-- [azure-cosmos](./sdk/cosmos/azure-cosmos)
-- [azure-eventhubs](./sdk/eventhub/azure-eventhubs)
+The libraries released in the November 2019 GA release:
 - [azure-keyvault-keys](./sdk/keyvault/azure-keyvault-keys)
 - [azure-keyvault-secrets](./sdk/keyvault/azure-keyvault-secrets)
 - [azure-identity](./sdk/identity/azure-identity)
 - [azure-storage-blob](./sdk/storage/azure-storage-blob)
-- [azure-storage-file](./sdk/storage/azure-storage-file-share)
 - [azure-storage-queue](./sdk/storage/azure-storage-queue)
 
->NOTE: If you need to ensure your code is ready for production use one of the stable libraries.
+The libraries released in the November 2019 preview:
+- [azure-cosmos](./sdk/cosmos/azure-cosmos)
+- [azure-eventhubs](./sdk/eventhub/azure-eventhubs)
+- [azure-storage-file-share](./sdk/storage/azure-storage-file-share)
 
+> NOTE: If you need to ensure your code is ready for production use one of the stable, non-preview libraries.
 
-### Client: Stable
-Last stable versions of packages that have been provided for usage with Azure and are production-ready. These libraries provide you with similar functionalities to the Preview ones as they allow you to use and consume existing resources and interact with them, for example: upload a blob.
+### Client: Previous Versions
+Last stable versions of packages that have been provided for usage with Azure and are production-ready. These libraries provide you with similar functionalities to the Preview ones as they allow you to use and consume existing resources and interact with them, for example: upload a blob. They might not implement the [guidelines](https://azure.github.io/azure-sdk/python_introduction.html) or have the same feature set as the Novemeber releases. They do however offer wider coverage of services. 
 
 ### Management
 Libraries which enable you to provision specific resources. They are responsible for directly mirroring and consuming Azure service's REST endpoints. The management libraries use the `azure-mgmt-<service name>` convention for their package names.
@@ -50,6 +50,10 @@ Libraries which enable you to provision specific resources. They are responsible
 * File an issue via [Github Issues](../../issues)
 * Check [previous questions](https://stackoverflow.com/questions/tagged/azure+python) or ask new ones on StackOverflow using `azure` and `python` tags.
 
+### Reporting security issues and security bugs
+
+Security issues and bugs should be reported privately, via email, to the Microsoft Security Response Center (MSRC) <secure@microsoft.com>. You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Further information, including the MSRC PGP key, can be found in the [Security TechCenter](https://www.microsoft.com/msrc/faqs-report-an-issue).
+
 ## Contributing
 For details on contributing to this repository, see the [contributing guide](CONTRIBUTING.md).
 

samples/smoketest/key_vault_certificates.py → common/smoketest/key_vault_certificates.py

@@ -22,13 +22,14 @@ def __init__(self):
         self.certificate_name = "cert-name-" + uuid.uuid1().hex
 
     def create_certificate(self):
-        self.certificate_client.begin_create_certificate(name=self.certificate_name, policy=CertificatePolicy.get_default()).wait()
+        print("creating certificate...")
+        self.certificate_client.create_certificate(name=self.certificate_name)
         print("\tdone")
 
     def get_certificate(self):
         print("Getting a certificate...")
-        certificate = self.certificate_client.get_certificate(name=self.certificate_name)
-        print(f"\tdone, certificate: {certificate.name}.")
+        certificate = self.certificate_client.get_certificate_with_policy(name=self.certificate_name)
+        print("\tdone, certificate: %s." % certificate.name)
 
     def delete_certificate(self):
         print("Deleting a certificate...")

samples/smoketest/key_vault_certificates_async.py → common/smoketest/key_vault_certificates_async.py

@@ -23,12 +23,13 @@ def __init__(self):
         self.certificate_name = "cert-name-" + uuid.uuid1().hex
 
     async def create_certificate(self):
-        await self.certificate_client.create_certificate(name=self.certificate_name, policy=CertificatePolicy.get_default())
+        create_certificate_poller = await self.certificate_client.create_certificate(name=self.certificate_name)
+        await create_certificate_poller
         print("\tdone")
 
     async def get_certificate(self):
         print("Getting a certificate...")
-        certificate = await self.certificate_client.get_certificate(name=self.certificate_name)
+        certificate = await self.certificate_client.get_certificate_with_policy(name=self.certificate_name)
         print(f"\tdone, certificate: {certificate.name}.")
 
     async def delete_certificate(self):

samples/smoketest/key_vault_keys.py → common/smoketest/key_vault_keys.py

@@ -23,17 +23,17 @@ def __init__(self):
 
     def create_rsa_key(self):
         print("Creating an RSA key...")
-        self.key_client.create_rsa_key(name=self.key_name, size=2048)
+        self.key_client.create_rsa_key(name=self.key_name, size=2048, hsm=False)
         print("\tdone")
 
     def get_key(self):
         print("Getting a key...")
         key = self.key_client.get_key(name=self.key_name)
-        print(f"\tdone, key: {key.name}.")
+        print("\tdone, key: %s." % key.name)
 
     def delete_key(self):
         print("Deleting a key...")
-        deleted_key = self.key_client.begin_delete_key(name=self.key_name).result()
+        deleted_key = self.key_client.delete_key(name=self.key_name)
         print("\tdone: " + deleted_key.name)
 
     def run(self):

samples/smoketest/key_vault_keys_async.py → common/smoketest/key_vault_keys_async.py

@@ -23,7 +23,7 @@ def __init__(self):
 
     async def create_rsa_key(self):
         print("Creating an RSA key...")
-        await self.key_client.create_rsa_key(name=self.key_name, size=2048)
+        await self.key_client.create_rsa_key(name=self.key_name, size=2048, hsm=False)
         print("\tdone")
 
     async def get_key(self):

common/smoketest/program.py

@@ -2,8 +2,12 @@
 # Copyright (c) Microsoft Corporation.
 # Licensed under the MIT License.
 # ------------------------------------
+import logging
+logging.basicConfig()
+
 import smoke_test
 
+
 try:
     import smoke_test_async
 except SyntaxError:

common/smoketest/requirements.txt

@@ -3,6 +3,8 @@ azure-core==1.0.0b3
 azure-cosmos==4.0.0b2
 azure-eventhub==5.0.0b3
 azure-identity==1.0.0b3
+azure-keyvault-certificates==4.0.0b3
+azure-keyvault-keys==4.0.0b3
 azure-keyvault-secrets==4.0.0b3
 azure-storage-blob==12.0.0b3
 azure-storage-common==2.1.0

common/smoketest/smoke_test.py

@@ -2,6 +2,8 @@
 # Copyright (c) Microsoft Corporation.
 # Licensed under the MIT License.
 # ------------------------------------
+from key_vault_certificates import KeyVaultCertificates
+from key_vault_keys import KeyVaultKeys
 from key_vault_secrets import KeyVaultSecrets
 from storage_blob import StorageBlob
 from event_hubs import EventHub
@@ -12,6 +14,8 @@
 print("      AZURE TRACK 2 SDKs SMOKE TEST")
 print("==========================================")
 
+KeyVaultCertificates().run()
+KeyVaultKeys().run()
 KeyVaultSecrets().run()
 StorageBlob().run()
 EventHub().run()

common/smoketest/smoke_test_async.py

@@ -3,6 +3,8 @@
 # Licensed under the MIT License.
 # ------------------------------------
 import asyncio
+from key_vault_certificates_async import KeyVaultCertificates
+from key_vault_keys_async import KeyVaultKeys
 from key_vault_secrets_async import KeyVaultSecrets
 from event_hubs_async import EventHub
 
@@ -13,6 +15,8 @@
 
 
 async def main():
+    await KeyVaultCertificates().run()
+    await KeyVaultKeys().run()
     await KeyVaultSecrets().run()
     await EventHub().run()
 

doc/sphinx/package_service_mapping.json

@@ -883,6 +883,11 @@
     "service_name": "Storage",
     "manually_generated": true
   },
+  "azure-storage-file-datalake": {
+    "category": "Client",
+    "service_name": "Storage",
+    "manually_generated": true
+  },
   "azure-storage-queue": {
     "category": "Client",
     "service_name": "Storage",

eng/pipelines/autorest_checks.yml

@@ -0,0 +1,50 @@
+# This pipeline is triggered only for pr
+trigger: none
+
+pr:
+  branches:
+    include:
+    - master
+    - feature/*
+    - hotfix/*
+    - release/*
+    - restapi*
+  paths:
+    include:
+    - sdk/core/
+
+variables:
+  NodeVersion: '10.x'
+  PythonVersion: '3.6'
+  auto_rest_clone_url: 'https://github.com/Azure/autorest.python.git'
+  repo_branch: 'azure-core'
+  source_path: 'sdk/core/azure-core'
+
+jobs:
+  - job: 'Run_AutoRest'
+    displayName: 'Run AutoRest'
+
+    pool:
+      vmImage: 'ubuntu-16.04'
+
+    steps:
+      - task: NodeTool@0
+        displayName: 'Install Node.js $(NodeVersion)'
+        inputs:
+          versionSpec: '$(NodeVersion)'        
+
+      - task: UsePythonVersion@0
+        displayName: 'Use Python $(PythonVersion)'
+        inputs:
+          versionSpec: $(PythonVersion)
+
+      - script: pip install pytest pytest-cov pytest-asyncio isodate msrest aiohttp $(Build.SourcesDirectory)/$(source_path)
+        displayName: 'Prepare Environment'
+
+      - script: |
+          git clone --recursive $(auto_rest_clone_url)
+          cd autorest.python
+          git checkout $(repo_branch)
+          npm install
+          pytest test/vanilla
+        displayName: 'Run Test'

eng/pipelines/smoke-test.yml

@@ -5,8 +5,8 @@ jobs:
   - job:
     strategy:
       matrix:
-        Python_374:
-          PythonVersion: '3.7.4'
+        Python_37:
+          PythonVersion: '3.7'
         Python_27:
           PythonVersion: '2.7'
           InstallAsyncRequirements: false

eng/pipelines/templates/jobs/archetype-sdk-client.yml

@@ -105,13 +105,13 @@ jobs:
               targetPath: $(Build.ArtifactStagingDirectory)
 
   - job: 'Test_Linux_Python38'
-    condition: ne(variables['Skip.Test'], 'true')
+    condition: and(succeededOrFailed(), ne(variables['Skip.Test'], 'true'))
     timeoutInMinutes: ${{ parameters.TestTimeoutInMinutes }}
     variables:
     - template: ../variables/globals.yml
 
     dependsOn:
-       - 'Build'
+      - 'Build'
 
     pool:
       vmImage: 'ubuntu-16.04'

eng/tox/import_all.py

@@ -18,9 +18,6 @@
 excluded_packages = [
     "azure",
     "azure-mgmt",
-    "azure.core.tracing.opencensus",
-    "azure.eventhub.checkpointstoreblob.aio",
-    "azure.storage.file.share" # Github issue 7879.
     ]
 
 def should_run_import_all(package_name):
@@ -41,17 +38,16 @@ def should_run_import_all(package_name):
 
     # get target package name from target package path
     pkg_dir = os.path.abspath(args.target_package)
-    pkg_name, _ = get_package_details(os.path.join(pkg_dir, 'setup.py'))
-    package_name = pkg_name.replace("-", ".")
+    package_name, namespace, _ = get_package_details(os.path.join(pkg_dir, 'setup.py'))
 
     if should_run_import_all(package_name):
         # import all modules from current package
         logging.info(
-            "Importing all modules from package [{0}] to verify dependency".format(
-                package_name
+            "Importing all modules from namespace [{0}] to verify dependency".format(
+                namespace
             )
         )
-        import_script_all = "from {0} import *".format(package_name)
+        import_script_all = "from {0} import *".format(namespace)
         exec(import_script_all)
         logging.info("Verified module dependency, no issues found")
     else:

eng/tox/prep_sphinx_env.py

@@ -44,12 +44,6 @@
 root_dir = os.path.abspath(os.path.join(os.path.abspath(__file__), "..", "..", ".."))
 sphinx_conf = os.path.join(root_dir, "doc", "sphinx", "individual_build_conf.py")
 
-# reference issue 8523 for eliminating this ridiculousness.
-UNFRIENDLY_PACKAGE_TO_NAMESPACE = {
-    'azure-storage-file-share': 'azure.storage.fileshare',
-    'azure-core-tracing-opencensus': 'azure.core.tracing.ext.opencensus_span',
-    'azure-eventhub-checkpointstoreblob-aio': 'azure.eventhub.extensions.checkpointstoreblobaio'
-}
 
 def should_build_docs(package_name):
     return not ("nspkg" in package_name or package_name in ["azure", "azure-mgmt", "azure-keyvault", "azure-documentdb", "azure-mgmt-documentdb", "azure-servicemanagement-legacy"])
@@ -98,10 +92,10 @@ def copy_conf(doc_folder):
     shutil.copy(sphinx_conf, os.path.join(doc_folder, 'conf.py'))
 
 
-def create_index(doc_folder, source_location, package_name):
+def create_index(doc_folder, source_location, namespace):
     index_content = ""
 
-    package_rst = "{}.rst".format(package_name.replace("-", "."))
+    package_rst = "{}.rst".format(namespace)
     content_destination = os.path.join(doc_folder, "index.rst")
 
     if not os.path.exists(doc_folder):
@@ -117,7 +111,7 @@ def create_index(doc_folder, source_location, package_name):
     elif rst_readmes:
         index_content = create_index_file(rst_readmes[0], package_rst)
     else:
-        logging.warning("No readmes detected for this package {}".format(package_name))
+        logging.warning("No readmes detected for this namespace {}".format(namespace))
         index_content = RST_EXTENSION_FOR_INDEX.format(package_rst)
 
     # write index
@@ -158,18 +152,15 @@ def write_version(site_folder, version):
     args = parser.parse_args()
 
     package_path = os.path.abspath(args.target_package)
-    package_name, package_version = get_package_details(
+    package_name, namespace, package_version = get_package_details(
         os.path.join(package_path, "setup.py")
     )
 
-    if package_name in UNFRIENDLY_PACKAGE_TO_NAMESPACE.keys():
-        package_name = UNFRIENDLY_PACKAGE_TO_NAMESPACE[package_name]
-
     if should_build_docs(package_name):
         source_location = move_and_rename(unzip_sdist_to_directory(args.dist_dir))
         doc_folder = os.path.join(source_location, "docgen")
 
-        create_index(doc_folder, source_location, package_name)
+        create_index(doc_folder, source_location, namespace)
 
         site_folder = os.path.join(args.dist_dir, "site")
         write_version(site_folder, package_version)