[FEAT] Infer Azure storage account from uri

[kevinzwang]

Resolves #3142

This PR also adds a dependency for the derive_builder crate, which I'm hoping to introduce into our repo and use throughout our code to reduce builder boilerplate.

[kevinzwang]

@anilmenon14 could you check if the changes in this PR works? Let me know if you need any help with that

[codspeed-hq]

CodSpeed Performance Report

Merging #3165 will not alter performance

_{Comparing kevin/azure-infer-storage-account (218d55f) with main (3cef614)}

Summary

✅ 17 untouched benchmarks

[codecov]

Codecov Report

Attention: Patch coverage is 30.15873% with 44 lines in your changes missing coverage. Please review.

Project coverage is 78.99%. Comparing base (3cef614) to head (218d55f).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
src/daft-io/src/azure_blob.rs	0.00%	43 Missing ⚠️
src/daft-io/src/lib.rs	93.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3165      +/-   ##
==========================================
- Coverage   79.00%   78.99%   -0.02%     
==========================================
  Files         634      634              
  Lines       76943    76958      +15     
==========================================
+ Hits        60789    60790       +1     
- Misses      16154    16168      +14     

Files with missing lines	Coverage Δ
src/daft-io/src/python.rs	87.09% <100.00%> (ø)
src/daft-io/src/lib.rs	72.19% <93.33%> (ø)
src/daft-io/src/azure_blob.rs	0.00% <0.00%> (ø)

... and 1 file with indirect coverage changes

[anilmenon14]

Hi @kevinzwang , the changes look great to me! I did a quick build of this branch on my laptop and ran some tests and they passed.
After the make build and make .venv steps on this branch, I ran the below commands in the Python REPL:

# This is the ideal set of steps for Azure
import os
import daft # Using version 0.3.9
from daft.unity_catalog import UnityCatalog

DATABRICKS_HOST = os.environ.get('DATABRICKS_HOST')
PAT_TOKEN_AZURE = os.environ.get('PAT_TOKEN_AZURE') 

unity = UnityCatalog(endpoint=DATABRICKS_HOST,token=PAT_TOKEN_AZURE)
unity_table_ext = unity.load_table("ucx_migrated.ucx_dst_external.external_dbfs_mounted") # This is an external table
df_ext = daft.read_deltalake(unity_table_ext)
df_ext.show()

This worked perfectly,by materializing the expected results of the Dataframe! 🙌

I also did a quick regression test using an AWS environment to confirm there has been no unintended impact seen elsewhere and it still works well there too.
If I can help with any additional manual testing on Unity Catalog , before merging, feel free to let me know and I can definitely do that.

[kevinzwang]

@anilmenon14 thanks for testing it! One final thing I'd like to confirm is that with your setup, the script fails without these changes, AKA that the Azure storage account was not set elsewhere in your testing, and that it is properly determined by Daft with this PR.

[anilmenon14]

Hi @kevinzwang , your statements are correct. With the code in this PR, run on a shell session without any pre-set environment variables for the Azure storage account, it is now able to determine the storage account.
Just to be extra sure, I did a comparison test with a virtual environment with the code from this PR versus the v0.3.9 on separate shell sessions initiated with same environment variables of access token and Host URL.
The code snippet only successfully works for the DataFrame .show() API in the code version from this PR.

[kevinzwang]

That's great to hear! Thank you again for such thorough testing. @raunakab please give this a review whenever you can

[raunakab]

Looks good to me!