Azure Blob - collectstatic not working

[raucodes]

Hi,

my running project is not working:

• Django 4.0rc1
• Django-storages 1.12.3

Requirements:

    django >= 4.0rc1
    mysqlclient
    django-froala-editor
    pillow
    stripe
    django-storages[azure]

custom_azure.py:

from storages.backends.azure_storage import AzureStorage

class AzureMediaStorage(AzureStorage):
    account_name = 'foo'  # Must be replaced by your <storage_account_name>
    account_key = 'zzzz=' # Must be replaced by your <storage_account_key>
    azure_container = 'media'
    expiration_secs = None
    overwrite_files = True

class AzureStaticStorage(AzureStorage):
    account_name = 'foo'  # Must be replaced by your storage_account_name
    account_key = 'zzzz=' # Must be replaced by your <storage_account_key>
    azure_container = 'static'
    expiration_secs = None
    overwrite_files = True

Error output:

Traceback (most recent call last):
  File "/mnt/c/Program Files/JetBrains/PyCharm 2021.2.3/plugins/python/helpers/pycharm/django_manage.py", line 52, in <module>
    run_command()
  File "/mnt/c/Program Files/JetBrains/PyCharm 2021.2.3/plugins/python/helpers/pycharm/django_manage.py", line 46, in run_command
    run_module(manage_file, None, '__main__', True)
  File "/usr/lib64/python3.9/runpy.py", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib64/python3.9/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/westcoast-dk/PycharmProjects/WestcoastShop/WestcoastShop/manage.py", line 24, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 425, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 373, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.9/site-packages/django/core/management/base.py", line 417, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 187, in handle
    collected = self.collect()
  File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
    handler(path, prefixed_path, storage)
  File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 334, in copy_file
    if not self.delete_file(path, prefixed_path, source_storage):
  File "/usr/local/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 248, in delete_file
    if self.storage.exists(prefixed_path):
  File "/usr/local/lib/python3.9/site-packages/storages/backends/azure_storage.py", line 241, in exists
    blob_client.get_blob_properties()
  File "/usr/local/lib/python3.9/site-packages/azure/core/tracing/decorator.py", line 83, in wrapper_use_tracer
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/azure/storage/blob/_blob_client.py", line 1242, in get_blob_properties
    process_storage_error(error)
  File "/usr/local/lib/python3.9/site-packages/azure/storage/blob/_shared/response_handlers.py", line 177, in process_storage_error
    exec("raise error from None")   # pylint: disable=exec-used # nosec
  File "<string>", line 1, in <module>
azure.core.exceptions.ClientAuthenticationError: Operation returned an invalid status 'Forbidden'
ErrorCode:AuthenticationFailed

Process finished with exit code 1

I don't have changed anything on the storage account or the settings.py

regards
Christopher

[jschneier]

Was there any one thing you changed in particular?

That is, what was the configuration when it was working and what did you then subsequently change? The only way to debug things is to perturb a single thing at a time.

[raucodes]

I just started the project and added a new logo, and then started the collectstatic command

Edit:

Ok if I change my settings.py:

from:

AZURE_CUSTOM_DOMAIN = f'files.foo.com'

back to:

AZURE_CUSTOM_DOMAIN = f'{AZURE_ACCOUNT_NAME}.blob.core.windows.net'

It is working, but with the custom domain it always works.

Regards
Christopher.

[dimbleby]

it's unclear which case is problematic - above you say that both cases are working, but I suppose you meant that one of them is not?

What changed since #1082 (comment)?

[raucodes]

Original which worked in the last 1,5 years but not anymore:

AZURE_CUSTOM_DOMAIN = f'files.foo.com'

New:

AZURE_CUSTOM_DOMAIN = f'{AZURE_ACCOUNT_NAME}.blob.core.windows.net'

is working for collectstatic command, but I want to use the custom domain.

Why is that used for authentication and not the AZURE_ACCOUNT_NAME

[raucodes]

Original which worked in the last 1,5 years but not anymore:

AZURE_CUSTOM_DOMAIN = f'files.foo.com'

New:

AZURE_CUSTOM_DOMAIN = f'{AZURE_ACCOUNT_NAME}.blob.core.windows.net'

is working for collectstatics , but I want to use the custom domain, why is that used for authentication and not the AZURE_ACCOUNT_NAME

[dimbleby]

#1083 explicitly uses the account name for authentication. Isn't this what you confirmed was working in that comment at #1082?

If there's still a problem here it might be best for you to figure out what the authentication to Azure needs to look like yourself: presumably you have a storage account set up with a custom domain, it shouldn't be too hard to experiment with ways of creating a BlobServiceClient and find out what works.

No doubt an MR would be welcome!

[raucodes]

This was another error message in #1082

... azure/storage/blob/_shared/base_client.py", line 351, in _format_shared_key_credential raise ValueError("Unable to determine account name for shared key credential.")

where the site was not loading the content in the frontend.

If I now keep it like:

AZURE_CUSTOM_DOMAIN = f'files.foo.com'

the content loads fine in the frontend and the src url is the right one.
But collectstatic says

azure.core.exceptions.ClientAuthenticationError: Operation returned an invalid status 'Forbidden' ErrorCode:AuthenticationFailed

It is working with the .blob.core.windows.net domain, but then all src urls are also from the .blob.core.windows.net domain.
It is the only problem with customdomain files.foo.com I can also upload pictures in my backend to files.foo.com but collectstatic is just working with the original storageaccount url.

[raucodes]

Ok the problem is that my "custom domain" is mapped via CDN and HTTPS but this will not be used for collectstatic, if I add a "custom domain" in the Azure Portal it is only http, but collectstatic works.

https://docs.microsoft.com/da-dk/azure/storage/blobs/storage-custom-domain-name?tabs=azure-portal
https://docs.microsoft.com/da-dk/azure/storage/blobs/storage-custom-domain-name?tabs=azure-portal#enable-https

For now, I will uncomment #AZURE_CUSTOM_DOMAIN = f'{AZURE_ACCOUNT_NAME}.blob.core.windows.net' if I upload static files, and for productive I use AZURE_CUSTOM_DOMAIN = 'files.foo.com' until I found a solution why the CDN custom domain will not be accepted for authentication

[jeffreykirchner]

I a having the same issue as @raucodes, collect static works if "AZURE_CUSTOM_DOMAIN" is commented out. Returns this error if not:

azure.core.exceptions.ClientAuthenticationError: Operation returned an invalid status 'Forbidden'
ErrorCode:AuthenticationFailed

My config:

app_name/custom_azure.py:

from storages.backends.azure_storage import AzureStorage

class AzureMediaStorage(AzureStorage):
    location = 'media'

class AzureStaticStorage(AzureStorage):
    location = 'static'

settings.py:

STATICFILES_STORAGE = 'app_name.custom_azure.AzureStaticStorage'
DEFAULT_FILE_STORAGE = 'app_name.custom_azure.AzureMediaStorage'

AZURE_CONTAINER = 'container_name'
AZURE_ACCOUNT_NAME = 'azure_storage_account_name'
AZURE_ACCOUNT_KEY =  'azure_storage_account_key'
AZURE_CUSTOM_DOMAIN = 'foo.azureedge.net'
AZURE_OVERWRITE_FILES = True

[milopersic]

Also having the same issue. Any updates on this? If I set AZURE_CUSTOM_DOMAIN to my CDN endpoint, uploads are broken. If I set the same to my storage name, uploads to my container happen but then files aren't served from the CDN.