Disable gzip auto decoding

[nitely]

Which service(blob, file, queue) does this issue concern?

blob

Which version of the SDK was used? Please provide the output of pip freeze.

Azure 4.0.0, blob 1.4.0

What problem was encountered?

When downloading a gzipped file that contains Content-Encoding: gzip through get_blob_to_stream, the file is automatically decoded. This was reported at #509. However, I find the workaround not to be maintainable since it messes with private method/classes. I believe this is a valid use-case, one may want a file to be decoded by the browser but not when fetching it through the API.

Have you found a mitigation/solution?

No

Note: for table service, please post the issue here instead: https://github.com/Azure/azure-cosmosdb-python.

[zezha-msft]

Hi @nitely, thanks for reaching out!

I really appreciate your feedback. This is however a breaking change. We'll consider including it as part of a redesign effort.

[markledwich2]

I just hit this more than 1 year after it was reported. Cost me some time and patience 👎

[MinhTraHong]

I'm waiting, too. The auto decode feature is really bad. We must change my all websites to AWS S3 which support gzip file with minimum transfer and high performance. I cannot think the reason why Azure do that.

[nitely]

We must change my all websites to AWS S3 which support gzip file with minimum transfer and high performance.

The file is decoded after it's downloaded. Performance is not really affected here. Azure does support gzip, the issue is just the python API doing this auto decoding which is quite unexpected.

[zezha-msft]

@xiafu-msft We have an open item for track 2 linked here.

Has it been done yet? We didn't do this change in track 1 due to breaking behavior.

[xiafu-msft]

Hi @zezha-msft
Here's the issue created by azure core Azure/azure-sdk-for-python#17145 after they support this we will enable user to disable auto gzip.

[vxfield]

For now, what I'm doing is directly downloading the blob using the rest API with a SAS key.

Potential workaround

1. First generate a blob URL containing the SAS key (there might be a more straightforward way to do this)

from azure.storage.blob import BlobServiceClient, generate_blob_sas, BlobSasPermissions
from datetime import datetime, timedelta

blob_service_client = BlobServiceClient.from_connection_string(connect_str)
blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
account_name = blob_service_client.credential.account_name
account_key = blob_service_client.credential.account_key
permission = BlobSasPermissions(read=True)
expiry = datetime.utcnow() + timedelta(hours=1)

sas_url =  blob_client.url + "?" + generate_blob_sas(account_name=account_name, container_name=container_name, blob_name=blob_name, account_key=account_key, permission=permission, expiry=expiry)

2. Directly download the blob content using some Python mechanism like the one below:

with open(file_path, 'wb') as f:
    with urllib.request.urlopen(sas_url) as response:
        file_size = int(response.getheader("Content-Length"))
        print("Downloading: %s Bytes: %s" % (file_path, file_size))

        downloaded_size = 0
        chunk_size = 4*1024*1024 # 4Mb chunk 
        while True:
            buffer = response.read(chunk_size)
            if not buffer:
                break

            downloaded_size += len(buffer)
            f.write(buffer)
            status = r"%10d  [%3.2f%%]" % (downloaded_size, downloaded_size * 100. / file_size)
            status = status + chr(8)*(len(status)+1)
            print(status)

Full example

Pre-reqs

1. Install the azure-storage-blob package (I tested with version 12.1.0)
2. Set AZURE_STORAGE_CONNECTION_STRING environment variable with your connection string

pip install azure-storage-blob
$env:AZURE_STORAGE_CONNECTION_STRING = "DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net"

Create file sample.py

from datetime import datetime, timedelta
import os, uuid
import gzip
import random
import string
import hashlib
from azure.storage.blob import BlobServiceClient, generate_blob_sas, BlobSasPermissions, __version__, ContentSettings
import urllib.request 

def _write_random_file(file_path, size):
    chars = ''.join([random.choice(string.ascii_letters) for i in range(size)])
    with open(file_path, 'w') as file_out:
        file_out.write(chars)

def _get_file_md5(file_path):
    with open(file_path, 'rb') as file_in:
        data = file_in.read()    
        md5 = hashlib.md5(data).hexdigest()
        return md5

def _zip_file(from_file_path, to_file_path):
    with open(from_file_path, 'rb') as file_in:
        with gzip.open(to_file_path, 'wb') as file_out:
            file_out.writelines(file_in)


def _unzip_file(from_file_path, to_file_path):
    with gzip.open(from_file_path, 'rb') as file_in:
        with open(to_file_path, 'wb') as file_out:
            file_out.writelines(file_in)


def _upload_blob(container_name, blob_name, file_path, content_type, content_encoding):
    print("\nUploading to Azure Storage as blob:\n\t" + blob_name)
    blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
    content_settings = ContentSettings(content_type=content_type, content_encoding=content_encoding)
    with open(file_path, "rb") as data:
        blob_client.upload_blob(data, content_settings=content_settings)


def _download_blob(container_name, blob_name, file_path):
    print("\nDownloading from Azure Storage blob:\n\t" + blob_name)
    # Instead of using the blob client to download the blob, we are going to use a 
    # custom solution
    #
    # blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
    # with open(file_path, "wb") as download_file:
    #    download_file.write(blob_client.download_blob(encoding="gzip").readall())
    #

    blob_service_client = BlobServiceClient.from_connection_string(connect_str)
    blob_client = blob_service_client.get_blob_client(container=container_name, blob=blob_name)
    account_name = blob_service_client.credential.account_name
    account_key = blob_service_client.credential.account_key
    permission = BlobSasPermissions(read=True)
    expiry = datetime.utcnow() + timedelta(hours=1)
    sas_url =  blob_client.url + "?" + generate_blob_sas(account_name=account_name, container_name=container_name, blob_name=blob_name, account_key=account_key, permission=permission, expiry=expiry)

    with open(file_path, 'wb') as f:
        with urllib.request.urlopen(sas_url) as response:
            file_size = int(response.getheader("Content-Length"))
            print("Downloading: %s Bytes: %s" % (file_path, file_size))

            downloaded_size = 0
            chunk_size = 4*1024*1024 # 4Mb chunk 
            while True:
                buffer = response.read(chunk_size)
                if not buffer:
                    break

                downloaded_size += len(buffer)
                f.write(buffer)
                status = r"%10d  [%3.2f%%]" % (downloaded_size, downloaded_size * 100. / file_size)
                status = status + chr(8)*(len(status)+1)
                print(status)


try:
    print("Azure Blob Storage v" + __version__)

    connect_str = os.getenv('AZURE_STORAGE_CONNECTION_STRING')

    # Create the BlobServiceClient object which will be used to create a container client
    blob_service_client = BlobServiceClient.from_connection_string(connect_str)

    # Create a unique name for the container
    container_name = str(uuid.uuid4())

    # Create the container
    container_client = blob_service_client.create_container(container_name)

    # Create a local directory to hold blob data
    local_path = "./data"
    os.makedirs(local_path, exist_ok=True)

    # Create a file in the local data directory to upload and download
    txt_file_name = str(uuid.uuid4()) + ".txt"
    txt_file_path = os.path.join(local_path, txt_file_name)
    zip_file_name = str.replace(txt_file_name, ".txt", ".gzip")
    zip_file_path = os.path.join(local_path, zip_file_name)

    # Write text to the file
    size = 20*1024*1024 # 20Mb file
    _write_random_file(file_path=txt_file_path, size=size)
    txt_md5 = _get_file_md5(file_path=txt_file_path)

    _zip_file(from_file_path=txt_file_path, to_file_path=zip_file_path)
    zip_md5 = _get_file_md5(file_path=zip_file_path)

    content_type = "text/plain" # could be json or anything else
    content_encoding = "gzip"
    _upload_blob(container_name=container_name, blob_name=zip_file_name, file_path=zip_file_path, content_type=content_type, content_encoding=content_encoding)

    downloaded_zip_file_path = os.path.join(local_path, str.replace(zip_file_name ,'.gzip', '.DOWNLOAD.gzip'))
    _download_blob(container_name=container_name, blob_name=zip_file_name, file_path=downloaded_zip_file_path)
    downloaded_zip_md5 = _get_file_md5(file_path=downloaded_zip_file_path)

    downloaded_txt_file_path = os.path.join(local_path, str.replace(txt_file_name ,'.txt', '.DOWNLOAD.txt'))
    try:
        _unzip_file(from_file_path=downloaded_zip_file_path, to_file_path=downloaded_txt_file_path)
    except:
        pass
    downloaded_txt_md5 = _get_file_md5(file_path=downloaded_txt_file_path)

    print("\nZip MD5           : " + zip_md5)
    print("Downloaded Zip MD5: " + downloaded_zip_md5)
    print(f'Zip MD5s are {"NOT " if zip_md5!=downloaded_zip_md5 else ""}equal')

    print("\nTxt MD5           : " + txt_md5)
    print("Downloaded Txt MD5: " + downloaded_txt_md5)
    print(f'Txt MD5s are {"NOT " if txt_md5!=downloaded_txt_md5 else ""}equal')

except Exception as ex:
    print('Exception:')
    print(ex)
    raise

Run

python ./sample.py

Expected result

The sample program will:

1. Create a container with a random name
2. Generate a 20Mb random text file
3. Compress the file with gzip
4. Upload the gzip file with gzip encoding to a (block) blob
5. Download the gzip file
6. Unzip the gzip file
7. Compare the MD5 hashes of the text and gzip files

Azure Blob Storage v12.1.0

Uploading to Azure Storage as blob:
        d99d6a73-5a81-4ef8-8549-d1f66185ef17.gzip

Downloading from Azure Storage blob:
        d99d6a73-5a81-4ef8-8549-d1f66185ef17.gzip
Downloading: ./data/d99d6a73-5a81-4ef8-8549-d1f66185ef17.DOWNLOAD.gzip Bytes: 15273869
   4194304  [27.46%]
   8388608  [54.92%]
  12582912  [82.38%]
  15273869  [100.00%]

Zip MD5           : 7936bf8e5676c5efc4fbcff341657168
Downloaded Zip MD5: 7936bf8e5676c5efc4fbcff341657168
Zip MD5s are equal

Txt MD5           : 2e30277eddd3c679dd9482378a9541f5
Downloaded Txt MD5: 2e30277eddd3c679dd9482378a9541f5
Txt MD5s are equal