Open
Description
Describe the bug
Following https://learn.microsoft.com/en-us/cli/azure/install-azure-cli-macos
For M1 macintosh az is not installed by brew in /opt/homebrew/bin/az per web instructions
I have Python 3.10.10 and az version 2.55 under
$HOMEDIR/miniconda3/bin as the base install before anything is defined.
I expected when I ran my command to upload-batch files from a local directory that the flag --if-unmodified-since would not go through the list of previously loaded files based on a timestamp that is one day past the last file transfer.
It appears that it is being ignored because it can't find the path to the models in Python 3.10 or 3.11 and I get variations on ModuleNotFoundError: No module named
For Python 3.10 I get ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2022_11_02.file'
For env
CONDA_PYTHON_EXE=$HOME/miniconda3/bin/python
PYTHONPATH=/opt/homebrew/bin/python3.10:$HOME/miniconda3/bin/python
I do have the directory under blob as $HOME/miniconda3/lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2022_11_02 -- see env and version info below.
Thanks,
Heidi
Related command
AZURE_CLI_DISABLE_CONNECTION_VERIFICATION=1; az storage blob upload-batch --destination $DESTDIR --account-name $STORAGE_BUCKET --source $SOURCE_DIR --sas-token "$SAS_TOKEN" --if-unmodified-since 2023-12-31T12:59:59Z --debug
Errors
ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2022_11_02.file'
and
The specified blob already exists.
RequestId:f992c4ad-b01e-0015-455a-427223000000
Time:2024-01-08T17:47:34.3068817Z
ErrorCode:BlobAlreadyExists
If you want to overwrite the existing one, please add --overwrite in your command.
Issue script & Debug output
I obscured the values for the flags, the hostname in the url, sas token and left the rest of the debug information.
az storage blob upload-batch --destination $DESTDIR --account-name $STORAGE_BUCKET --source $SOURCE_DIR --sas-token "$SAS_TOKEN" --if-unmodified-since 2023-12-31T12:59:59Z --debug
cli.knack.cli: Command arguments: ['storage', 'blob', 'upload-batch', '--destination', '$DESTDIR', '--account-name', '$STORAGE_BUCKET', '--source', '$SOURCEDIR', '--sas-token', '$SAS_TOKEN', '--if-unmodified-since', '2023-12-31T12:59:59Z', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x105320e50>, <function OutputProducer.on_global_arguments at 0x105490280>, <function CLIQuery.on_global_arguments at 0x1054cd510>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'storage': ['azext_storage_preview', 'azure.cli.command_modules.storage']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: storage 0.077 59 273
cli.azure.cli.core: Total (1) 0.077 59 273
cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
cli.azure.cli.core: Loading extensions:
cli.azure.cli.core: Name Load Time Groups Commands Directory
cli.knack.cli: Event: CommandLoader.OnLoadCommandTable []
cli.azure.cli.core: storage-preview 0.007 16 52 $HOME/.azure/cliextensions/storage-preview
cli.azure.cli.core: Total (1) 0.007 16 52
cli.azure.cli.core: Loaded 65 groups, 295 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : storage blob upload-batch
cli.azure.cli.core: Command table: storage blob upload
cli.azure.cli.core: remaining : batch
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x106325a20>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '$HOME/.azure/commands/2024-01-08.15-43-24.storage_blob_upload-batch.73143.log'.
az_command_data_logger: command args: storage blob upload-batch --destination {} --account-name {} --source {} --sas-token {} --if-unmodified-since {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x10633e7a0>]
cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "$HOMEDIR/miniconda3/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 655, in _get_attr
op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2023_01_01.models' has no attribute 'ActiveDirectoryPropertiesAccountType'
cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "$HOMEDIR/miniconda3/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 655, in _get_attr
op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2023_01_01.models' has no attribute 'ListKeyExpand'
cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "$HOMEDIR/miniconda3/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 655, in _get_attr
op = getattr(op, part)
AttributeError: module 'azure.mgmt.storage.v2023_01_01.models' has no attribute 'CorsRuleAllowedMethodsItem'
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x10638c790>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x10638c8b0>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x105490310>, <function CLIQuery.handle_query_parameter at 0x1054cd5a0>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x10638c820>]
Command group 'az storage' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
cli.azure.cli.command_modules.storage._validators: Try to get storage auth_mode value from environment variables or config file.
cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "$HOMEDIR/miniconda3/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 651, in _get_attr
op = import_module(full_mod_path)
File "$HOMEDIR/miniconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2022_11_02.blob'
cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "$HOMEDIR/miniconda3/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 651, in _get_attr
op = import_module(full_mod_path)
File "$HOMEDIR/miniconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2022_11_02.file'
cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "$HOMEDIR/miniconda3/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 651, in _get_attr
op = import_module(full_mod_path)
File "$HOMEDIR/miniconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2022_11_02.blob'
cli.azure.cli.core.profiles._shared: Traceback (most recent call last):
File "$HOMEDIR/miniconda3/lib/python3.10/site-packages/azure/cli/core/profiles/_shared.py", line 651, in _get_attr
op = import_module(full_mod_path)
File "$HOMEDIR/miniconda3/lib/python3.10/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'azure.multiapi.storagev2.blob.v2022_11_02.file'
cli.azure.cli.core._debug: Connection verification disabled by environment variable AZURE_CLI_DISABLE_CONNECTION_VERIFICATION
azure.multiapi.storagev2.blob.v2022_11_02._generated._serialization: Datetime with no tzinfo will be considered UTC.
urllib3.connectionpool: Starting new HTTPS connection (1): proxy:443
$HOMEDIR/miniconda3/lib/python3.10/site-packages/urllib3/connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'proxy'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
warnings.warn(
urllib3.connectionpool: https://proxy:443 "PUT $PATH-TO-FILE file1.name?$SAS_TOKEN HTTP/1.1" 412 252
cli.azure.cli.command_modules.storage.operations.blob: The specified blob already exists.
RequestId:bf86f837-a01e-0026-2673-422d88000000
Time:2024-01-08T20:43:24.9579532Z
ErrorCode:BlobAlreadyExists
If you want to overwrite the existing one, please add --overwrite in your command.
Expected behavior
The behavior I was hoping for was the command to return with no new files to be loaded and the previous loaded files would not be pre-processed again with ErrorCode:BlobAlreadyExists
Is there a test I can run to see if the erroring modules are imported properly?
Thanks! Heidi
Environment Summary
azure-cli 2.55.0
core 2.55.0
telemetry 1.1.0
Extensions:
storage-preview 1.0.0b1
Dependencies:
msal 1.24.0b2
azure-mgmt-resource 23.1.0b2
Python location '/opt/homebrew/Cellar/azure-cli/2.55.0/libexec/bin/python'
Extensions directory '/Users/heidischmidt/.azure/cliextensions'
Python (Darwin) 3.11.6 (main, Oct 2 2023, 13:45:54) [Clang 15.0.0 (clang-1500.0.40.1)]
Legal docs and information: aka.ms/AzureCliLegal
and in the miniconda base the pip list includes the following azure modules
azure-appconfiguration 1.1.1
azure-batch 14.0.0
azure-cli 2.55.0
azure-cli-core 2.55.0
azure-cli-telemetry 1.1.0
azure-common 1.1.28
azure-core 1.29.6
azure-cosmos 3.2.0
azure-data-tables 12.4.0
azure-datalake-store 0.0.53
azure-graphrbac 0.60.0
azure-identity 1.15.0
azure-keyvault-administration 4.4.0b2
azure-keyvault-certificates 4.7.0
azure-keyvault-keys 4.9.0b3
azure-keyvault-secrets 4.7.0
azure-loganalytics 0.1.1
azure-mgmt-advisor 9.0.0
azure-mgmt-apimanagement 4.0.0
azure-mgmt-appconfiguration 3.0.0
azure-mgmt-appcontainers 2.0.0
azure-mgmt-applicationinsights 1.0.0
azure-mgmt-authorization 4.0.0
azure-mgmt-batch 17.0.0
azure-mgmt-batchai 7.0.0b1
azure-mgmt-billing 6.0.0
azure-mgmt-botservice 2.0.0
azure-mgmt-cdn 12.0.0
azure-mgmt-cognitiveservices 13.5.0
azure-mgmt-compute 30.3.0
azure-mgmt-containerinstance 10.1.0
azure-mgmt-containerregistry 10.1.0
azure-mgmt-containerservice 28.0.0
azure-mgmt-core 1.4.0
azure-mgmt-cosmosdb 9.3.0
azure-mgmt-databoxedge 1.0.0
azure-mgmt-datalake-nspkg 3.0.1
azure-mgmt-datalake-store 0.5.0
azure-mgmt-datamigration 10.0.0
azure-mgmt-devtestlabs 4.0.0
azure-mgmt-dns 8.0.0
azure-mgmt-eventgrid 10.2.0b2
azure-mgmt-eventhub 10.1.0
azure-mgmt-extendedlocation 1.0.0b2
azure-mgmt-hdinsight 9.0.0
azure-mgmt-imagebuilder 1.2.0
azure-mgmt-iotcentral 10.0.0b2
azure-mgmt-iothub 3.0.0
azure-mgmt-iothubprovisioningservices 1.1.0
azure-mgmt-keyvault 10.3.0
azure-mgmt-kusto 0.3.0
azure-mgmt-loganalytics 13.0.0b4
azure-mgmt-managedservices 1.0.0
azure-mgmt-managementgroups 1.0.0
azure-mgmt-maps 2.0.0
azure-mgmt-marketplaceordering 1.1.0
azure-mgmt-media 9.0.0
azure-mgmt-monitor 5.0.1
azure-mgmt-msi 7.0.0
azure-mgmt-netapp 10.1.0
azure-mgmt-nspkg 3.0.2
azure-mgmt-policyinsights 1.1.0b4
azure-mgmt-privatedns 1.0.0
azure-mgmt-rdbms 10.2.0b13
azure-mgmt-recoveryservices 2.5.0
azure-mgmt-recoveryservicesbackup 7.0.0
azure-mgmt-redhatopenshift 1.4.0
azure-mgmt-redis 14.1.0
azure-mgmt-resource 23.1.0b2
azure-mgmt-search 9.1.0
azure-mgmt-security 5.0.0
azure-mgmt-servicebus 8.2.0
azure-mgmt-servicefabric 1.0.0
azure-mgmt-servicefabricmanagedclusters 1.0.0
azure-mgmt-servicelinker 1.2.0b1
azure-mgmt-signalr 2.0.0b1
azure-mgmt-sql 4.0.0b13
azure-mgmt-sqlvirtualmachine 1.0.0b5
azure-mgmt-storage 21.1.0
azure-mgmt-synapse 2.1.0b5
azure-mgmt-trafficmanager 1.0.0
azure-mgmt-web 7.0.0
azure-multiapi-storage 1.2.0
azure-nspkg 3.0.2
azure-storage-blob 12.19.0
azure-storage-common 1.4.2
azure-storage-file-share 12.15.0
azure-synapse-accesscontrol 0.5.0
azure-synapse-artifacts 0.17.0
azure-synapse-managedprivateendpoints 0.4.0
azure-synapse-spark 0.2.0
msrestazure 0.6.4
and in the Python 3.10 site-packages under storagev2/blob/v2022_11_02 directory
ls -l $HOME/miniconda3//lib/python3.10/site-packages/azure/multiapi/storagev2/blob/v2022_11_02/ []
total 1240
-rw-r--r-- 1 hes staff 9915 Jan 8 11:31 __init__.py
drwxr-xr-x 17 hes staff 544 Jan 8 11:31 __pycache__
-rw-r--r-- 1 hes staff 233131 Jan 8 11:31 _blob_client.py
-rw-r--r-- 1 hes staff 39496 Jan 8 11:31 _blob_service_client.py
-rw-r--r-- 1 hes staff 92743 Jan 8 11:31 _container_client.py
-rw-r--r-- 1 hes staff 8744 Jan 8 11:31 _deserialize.py
-rw-r--r-- 1 hes staff 32099 Jan 8 11:31 _download.py
-rw-r--r-- 1 hes staff 41014 Jan 8 11:31 _encryption.py
drwxr-xr-x 12 hes staff 384 Jan 8 11:31 _generated
-rw-r--r-- 1 hes staff 18749 Jan 8 11:31 _lease.py
-rw-r--r-- 1 hes staff 14741 Jan 8 11:31 _list_blobs_helper.py
-rw-r--r-- 1 hes staff 57319 Jan 8 11:31 _models.py
-rw-r--r-- 1 hes staff 6270 Jan 8 11:31 _quick_query_helper.py
-rw-r--r-- 1 hes staff 7918 Jan 8 11:31 _serialize.py
drwxr-xr-x 18 hes staff 576 Jan 8 11:31 _shared
-rw-r--r-- 1 hes staff 32372 Jan 8 11:31 _shared_access_signature.py
-rw-r--r-- 1 hes staff 13955 Jan 8 11:31 _upload_helpers.py
-rw-r--r-- 1 hes staff 331 Jan 8 11:31 _version.py
drwxr-xr-x 12 hes staff 384 Jan 8 11:31 aio
Additional context
No response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
