Azure Storage Sample Automation #39944

Yionse · 2025-03-05T02:33:59Z

By changing the current Storage samples, can enable them to run automatically in the pipeline.

For the latest pipeline results: https://dev.azure.com/azure-sdk/internal/_build/results?buildId=4713055&view=results

The supported packages are as follows:

azure-storage-blob
azure-storage-file-datalake
azure-storage-file-share
azure-storage-queue

The specific modifications are as follows:

Skipped azure-storage-blob/blob_samples_proxy_configuration.py because it uses a proxy and is not suitable for running on pipelines.
Change the environment variable name in samples to be consistent with that exported in test-resources.json.

Change the path of the local files used in the samples to an absolute address.
Update

SOURCE_FOLDER = "./sample-blobs/"

to

current_dir = os.path.dirname(os.path.abspath(__file__))
SOURCE_FOLDER = os.path.join(current_dir, "./sample-blobs/")

Comment out the following code snippet in blob_samples_container_access_policy.py, as setting the allowBlobPublicAccess attribute in test-resources.json is unstable and may not always take effect.
```
container_client.set_container_access_policy(signed_identifiers=identifiers, public_access=public_access)
```
Create required resources and add output in test-resources.json.

@xiangyan99 for notification.

azure-sdk · 2025-03-05T04:16:55Z

API change check

API changes are not detected in this pull request.

Copilot

Pull Request Overview

This PR automates Azure Storage samples for pipeline execution by standardizing environment variable names, adjusting local file paths to absolute paths, and commenting out unstable code sections.

Environment variables have been renamed from AZURE_STORAGE_* to STORAGE_* in most samples.
File paths have been converted from relative to absolute paths in several samples.
Unstable container access policy code has been commented out in async samples.

Reviewed Changes

Copilot reviewed 67 out of 67 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
sdk/storage/azure-storage-blob/samples/blob_samples_client_side_encryption_keyvault.py	Updated env var names and added key generation logic for client-side encryption.
sdk/storage/azure-storage-blob/samples/blob_samples_enumerate_blobs_async.py	Renamed environment variable and updated container name for async enumeration.
sdk/storage/azure-storage-blob/samples/blob_samples_enumerate_blobs.py	Renamed environment variable and updated container name for blob enumeration.
sdk/storage/azure-storage-blob/samples/blob_samples_directory_interface*.py	Renamed environment variable and hardcoded container name for directory interface samples.
scripts/devops_tasks/test_run_samples.py	Added proxy configuration sample to the list of skipped samples.
sdk/storage/azure-storage-blob/samples/blob_samples_container_access_policy_async.py	Renamed environment variable and commented out unstable access policy setting.
sdk/storage/azure-storage-blob/samples/blob_samples_copy_blob*.py	Updated env var names and container names; added container creation call.
sdk/storage/azure-storage-blob/samples/blob_samples_containers*.py	Renamed environment variable, updated file path for source file, and changed container names.
sdk/storage/azure-storage-blob/samples/blob_samples_authentication*	Updated environment variable names for connection string and account keys.
sdk/storage/azure-storage-blob/samples/blob_samples_client_side_encryption.py	Updated environment variable name in sample header and connection string retrieval.
sdk/storage/azure-storage-blob/samples/blob_samples_common*.py	Renamed env var and updated local file path for sample source file.

Comments suppressed due to low confidence (2)

sdk/storage/azure-storage-blob/samples/blob_samples_common.py:32

The environment variable used here is 'STORAGE_CONNECTION_STRING_SOFT', which is inconsistent with other samples that reference 'STORAGE_CONNECTION_STRING'. Please unify the variable naming.

connection_string = os.getenv("STORAGE_CONNECTION_STRING_SOFT")

sdk/storage/azure-storage-blob/samples/blob_samples_common_async.py:31

The use of 'STORAGE_CONNECTION_STRING_SOFT' here is inconsistent with other parts of the code that expect 'STORAGE_CONNECTION_STRING'. Consider using the same environment variable name across all samples.

connection_string = os.getenv("STORAGE_CONNECTION_STRING_SOFT")

sdk/storage/azure-storage-blob/samples/blob_samples_copy_blob.py

sdk/storage/azure-storage-blob/pyproject.toml

sdk/storage/azure-storage-blob/samples/blob_samples_client_side_encryption_keyvault.py

lmazuel · 2025-03-21T17:32:03Z

sdk/storage/azure-storage-blob/samples/blob_samples_container_access_policy_async.py

@@ -64,7 +63,7 @@ async def get_and_set_container_access_policy():
        public_access = PublicAccess.CONTAINER

        # Set the access policy on the container
-        await container_client.set_container_access_policy(signed_identifiers=identifiers, public_access=public_access)
+        # await container_client.set_container_access_policy(signed_identifiers=identifiers, public_access=public_access)


Why? Is this is not useful, remove. If this useful but can't be executed, then put extra context to explain to the customer why he may want to uncomment it

@lmazuel The storage account we created using arm template is somehow unstable (we tried to set its property allowblobpublicaccess as true, yet sometimes it works, sometimes it doesn't follow the template), that would cause failure to this sample. The sample itself does not have any issue, so is that ok to not make any changes on it but just to skip it in the pipeline?
f5a69f4#diff-f960dc3f6bb80a3355f294b17a15868dc7721a421cee036bf9f8cf2f4136bc58R158

@ChenxiJiang333 we have public blob disabled by policy in our test subscriptions which could be what you are seeing, unless you use our TME based subscription for testing: https://dev.azure.com/azure-sdk/internal/_wiki/wikis/internal.wiki/206/Subscription-and-Tenant-Usage?anchor=azure-sdk-test-resources-tme

I believe when they ran the sample in test pipeline, it was running under TME? @benbp

Blob Public Access is off by default for new accounts and is not turned on explicitly in our ARM template accounts because it is not allowed in Storage's test subscription. (We use TME for Live tests where it would be permitted but not for local testing).

I am okay if we leave the sample as-is and just ignore the samples that rely on public access as we do not want to enable that in our ARM template.

sdk/storage/azure-storage-blob/samples/blob_samples_client_side_encryption_keyvault.py

xiangyan99 · 2025-03-31T23:27:33Z

scripts/devops_tasks/test_run_samples.py

+        "blob_samples_proxy_configuration.py",
+        "blob_samples_container_access_policy.py",
+        "blob_samples_container_access_policy_async.py"
+    ],


Could you add some comments why we ignore these samples?

e.g. are they flaky code?

Proxy likely cannot be done in the pipeline, container access policies can be flakey but curious on what specifically is going wrong.

Ah, this is probably because of container public access? I am okay to ignore those.

Hi @xiangyan99, thanks for @jalauzon-msft's explanation, proxy cannot be done in the pipeline and somehow access policy samples are flakey possibly because of subscription policy. Should we put these comments into the code?

I don't think we should have anything commented in code. I think we should try and understand the flakiness and maybe address it, remove the sample entirely, or leave it disabled here.

Comment in the PR is good enough. :)

jalauzon-msft · 2025-03-31T23:52:34Z

sdk/storage/test-resources.json

+            "dependsOn": [
+                "[concat('Microsoft.Storage/storageAccounts/', variables('primaryAccountName'))]"
+            ]
+        },


Do we have to create containers in the ARM template? Could the samples create any containers they need to use so we can remove these?

Have removed them from template and create containers in the samples. Thanks!

jalauzon-msft · 2025-03-31T23:54:10Z

sdk/storage/azure-storage-blob/dev_requirements.txt

+aiohttp>=3.0
+azure-keyvault


I'd rather not add KeyVault as a dev dependency of Blob. Could we just ignore the one sample that uses KeyVault instead? That sample is for client-side encryption which is not a scenario many customers use and we do not encourage the use of it at all. I think it would be okay if that sample is ignored.

Hi @xiangyan99, is that ok to ignore this sample?

Yes. Let's remove it.

@xiangyan99 should we be clear in the sample that we don't encourage this scenario?

I would rather delete it than leaving the code and adding comments this is discouraged.

Leaving it may confuse both customers and AI.

As mentioned in the email thread, I would like to keep the sample, just disable it in test_run_samples.py so we can remove this dependency.

Discussed with Jacob offline. @ChenxiJiang333 let's remove the keyvault dependency and disable the samples that rely on keyvault. (Keep the sample but not run them in the pipeline)

Ok, updated the code.

jalauzon-msft · 2025-04-02T00:40:15Z

sdk/storage/azure-storage-blob/samples/blob_samples_client_side_encryption_keyvault.py

+KEYVAULT_URL = 'KEYVAULT_URL'
+AZURE_CLIENT_ID = 'ACTIVE_DIRECTORY_APPLICATION_ID'
+AZURE_CLIENT_SECRET = 'ACTIVE_DIRECTORY_APPLICATION_SECRET'
+AZURE_TENANT_ID = 'ACTIVE_DIRECTORY_TENANT_ID'


If we keep this sample, can we remove AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, and AZURE_TENANT_ID? I don't think they are used in the sample at all. I think we missed cleaning these up a while ago.

Has removed these unused codes in the sample, thanks!

jalauzon-msft

This looks good to me! Thanks for your work on this.

I will start a run of the test pipeline on this PR and assuming that looks good, this should be good to go.

jalauzon-msft · 2025-04-04T00:23:08Z

/azp run python - storage - tests

azure-pipelines · 2025-04-04T00:23:30Z

Azure Pipelines successfully started running 1 pipeline(s).

github-actions bot added the Storage Storage Service (Queues, Blobs, Files) label Mar 5, 2025

Yionse force-pushed the feat/automation_storage branch 2 times, most recently from 000080f to 6e560df Compare March 5, 2025 03:23

complete storage automation

e3c413c

Yionse force-pushed the feat/automation_storage branch 3 times, most recently from ecb648c to e3c413c Compare March 6, 2025 02:53

fix storage

91e72c1

Yionse force-pushed the feat/automation_storage branch 2 times, most recently from d79c18d to 91e72c1 Compare March 7, 2025 06:47

Yionse and others added 2 commits March 13, 2025 10:18

Merge branch 'Azure:main' into feat/automation_storage

e3b145d

fix

0a3561d

Yionse force-pushed the feat/automation_storage branch from 4cae2c5 to 0a3561d Compare March 14, 2025 02:10

Yionse marked this pull request as ready for review March 14, 2025 09:25

Copilot bot review requested due to automatic review settings March 14, 2025 09:25

Yionse requested review from annatisch, jalauzon-msft, vincenttran-msft and weirongw23-msft as code owners March 14, 2025 09:25

Copilot AI reviewed Mar 14, 2025

View reviewed changes

sdk/storage/azure-storage-blob/samples/blob_samples_copy_blob.py Show resolved Hide resolved

Create cspell.yaml

17f2c9c

Yionse requested review from scbedd, weshaggard and benbp as code owners March 17, 2025 06:27

ChenxiJiang333 force-pushed the feat/automation_storage branch from 82c810b to a1e4cce Compare March 17, 2025 09:58

lmazuel reviewed Mar 17, 2025

View reviewed changes

sdk/storage/azure-storage-blob/pyproject.toml Outdated Show resolved Hide resolved

skip mypy for specified line

b6292f7

ChenxiJiang333 force-pushed the feat/automation_storage branch from 2767729 to b6292f7 Compare March 18, 2025 04:25

weirongw23-msft reviewed Mar 18, 2025

View reviewed changes

sdk/storage/azure-storage-blob/samples/blob_samples_client_side_encryption_keyvault.py Outdated Show resolved Hide resolved

ChenxiJiang333 and others added 3 commits March 19, 2025 10:06

Add type annotations for key_bytes and key_id

7ff24cc

Fix container conflicts

3d87cdb

Fix type issues in KeyVaultKey initialization

3f0c14e

lmazuel reviewed Mar 21, 2025

View reviewed changes

mccoyp reviewed Mar 21, 2025

View reviewed changes

sdk/storage/azure-storage-blob/samples/blob_samples_client_side_encryption_keyvault.py Outdated Show resolved Hide resolved

v-xuto requested review from lmazuel and mccoyp March 24, 2025 05:31

Fix container conflicts

1a63e58

ChenxiJiang333 force-pushed the feat/automation_storage branch 2 times, most recently from e8d8727 to ff171e3 Compare March 24, 2025 07:25

ChenxiJiang333 added 2 commits March 24, 2025 15:27

Replace SecretClient with KeyClient in sample

f599b2e

skip container access policy samples

f5a69f4

ChenxiJiang333 force-pushed the feat/automation_storage branch from ff171e3 to f5a69f4 Compare March 24, 2025 07:28

xiangyan99 reviewed Mar 31, 2025

View reviewed changes

jalauzon-msft reviewed Mar 31, 2025

View reviewed changes

ChenxiJiang333 added 2 commits April 1, 2025 10:56

create containers in the samples

4212d6d

Handle existing container creation error

2e76409

jalauzon-msft reviewed Apr 2, 2025

View reviewed changes

remove keyvault dependency and skip keyvault sample tests

ee1c984

xiangyan99 approved these changes Apr 2, 2025

View reviewed changes

jalauzon-msft approved these changes Apr 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Azure Storage Sample Automation #39944

Azure Storage Sample Automation #39944

Yionse commented Mar 5, 2025 •

edited by v-xuto

Loading

azure-sdk commented Mar 5, 2025

Copilot AI left a comment

lmazuel Mar 21, 2025

ChenxiJiang333 Mar 24, 2025 •

edited

Loading

benbp Mar 27, 2025

xiangyan99 Mar 27, 2025

jalauzon-msft Apr 1, 2025

xiangyan99 Mar 31, 2025

jalauzon-msft Mar 31, 2025

jalauzon-msft Apr 1, 2025

ChenxiJiang333 Apr 1, 2025 •

edited

Loading

jalauzon-msft Apr 2, 2025

xiangyan99 Apr 2, 2025 •

edited

Loading

jalauzon-msft Mar 31, 2025

ChenxiJiang333 Apr 1, 2025

jalauzon-msft Mar 31, 2025

ChenxiJiang333 Apr 1, 2025

xiangyan99 Apr 1, 2025

lmazuel Apr 1, 2025

xiangyan99 Apr 1, 2025

jalauzon-msft Apr 2, 2025

xiangyan99 Apr 2, 2025

ChenxiJiang333 Apr 2, 2025

jalauzon-msft Apr 2, 2025

ChenxiJiang333 Apr 2, 2025

jalauzon-msft left a comment

jalauzon-msft commented Apr 4, 2025

azure-pipelines bot commented Apr 4, 2025

		aiohttp>=3.0
		azure-keyvault

Azure Storage Sample Automation #39944

Are you sure you want to change the base?

Azure Storage Sample Automation #39944

Conversation

Yionse commented Mar 5, 2025 • edited by v-xuto Loading

azure-sdk commented Mar 5, 2025

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Choose a reason for hiding this comment

ChenxiJiang333 Mar 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ChenxiJiang333 Apr 1, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiangyan99 Apr 2, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jalauzon-msft left a comment

Choose a reason for hiding this comment

jalauzon-msft commented Apr 4, 2025

azure-pipelines bot commented Apr 4, 2025

Yionse commented Mar 5, 2025 •

edited by v-xuto

Loading

ChenxiJiang333 Mar 24, 2025 •

edited

Loading

ChenxiJiang333 Apr 1, 2025 •

edited

Loading

xiangyan99 Apr 2, 2025 •

edited

Loading