Add Support for Microsoft OneLake

[djouallah]

Description

Microsoft recently released "in great fanfare" OneLake which is a managed lakehouse offering based on ADLS Gen2.

Reading works fine but writing generate an error, the Parquet is written but in the log, we get a tmp json file

using Polars writer which is based on Deltalake writer I believe get this error
ldf.write_delta("/lakehouse/default/Files/regionn")

PyDeltaTableError                         Traceback (most recent call last)
Cell In[26], line 1
----> 1 ldf.write_delta("/lakehouse/default/Files/regionn")

File ~/cluster-env/trident_env/lib/python3.10/site-packages/polars/dataframe/frame.py:3363, in DataFrame.write_delta(self, target, mode, overwrite_schema, storage_options, delta_write_options)
   3360     if data_schema == table_schema.to_pyarrow(as_large_types=True):
   3361         data_schema = table_schema.to_pyarrow()
-> 3363 write_deltalake(
   3364     table_or_uri=target,
   3365     data=data,
   3366     mode=mode,
   3367     schema=data_schema,
   3368     overwrite_schema=overwrite_schema,
   3369     storage_options=storage_options,
   3370     **delta_write_options,
   3371 )

File ~/cluster-env/trident_env/lib/python3.10/site-packages/deltalake/writer.py:322, in write_deltalake(table_or_uri, data, schema, partition_by, filesystem, mode, file_options, max_partitions, max_open_files, max_rows_per_file, min_rows_per_group, max_rows_per_group, name, description, configuration, overwrite_schema, storage_options, partition_filters)
    302 ds.write_dataset(
    303     data,
    304     base_dir="/",
   (...)
    318     max_partitions=max_partitions,
    319 )
    321 if table is None:
--> 322     _write_new_deltalake(
    323         table_uri,
    324         schema,
    325         add_actions,
    326         mode,
    327         partition_by or [],
    328         name,
    329         description,
    330         configuration,
    331         storage_options,
    332     )
    333 else:
    334     table._table.create_write_transaction(
    335         add_actions,
    336         mode,
   (...)
    339         partition_filters,
    340     )

PyDeltaTableError: Failed to read delta log object: Generic LocalFileSystem error: Unable to rename file: No such file or directory (os error 2)

[cnolanminich]

This is of great interest to my team as well -- the promise of OneLake as a managed access layer for a data lake (but being open to any compute engine to write) would be absolutely incredible!

[rtyler]

From the error and the fact that a temp file is remaining, it does look like the rename operation is failing. Can you share the URL scheme that the Delta table is being loaded with/ I'm wondering if Azure is exposing "raw" ADLSv2 or masking it somehow. This may require us to have a specific object_store implementation 🤔

[roeap]

It seems like accessing the underlying one lake storage should be compatible with adls gen2 apis, but require a quite different kind of url. https://learn.microsoft.com/en-us/fabric/onelake/onelake-access-api

It seems they are using the gen2 apis.. unfortunately right now object store always calls the blob apis which work with both gen1 and gen2 storages, but may not work with one lake, we would have to find out.

While the docs are very limited right now I have not found any blocking disparity between gen2 and one lake storage from the api side. The authorization docs are very vague, but it sounds like things may just work using the same tokens as storage accounts do.

So if we are lucky we just need to handle urls correctly, maybe we also need to reimplement some things to use gen2 style requests only, as one lake apis may no longer support blob api requests.

[mabasile-MSFT]

Hi all, I'm on the OneLake team. I was trying to repro this on my side and was running into a "URL did not match any known pattern for scheme" error - does DeltaLake perform URL validation checks and is blocking the 'onelake.dfs.fabric.microsoft.com' format?

Also, @roeap - right now you can use the format "onelake.blob.fabric.microsoft.com" for OneLake to support Blob API calls. However this isn't supported for calls to the ABFS driver, so I'm not sure if that will help you here.

[jocaplan]

It looks like there is some validation of endpoints for Azure.

Adding two more lines to the Azure section of the config might be enough. It would also need to support: dfs.fabric.microsoft.com and blob.fabric.microsoft.com.

delta-rs/rust/src/storage/config.rs

Line 124 in b17f286

|| host.contains("blob.core.windows.net")

If you are supporting only blob, you can replace dfs in the URL with blob which you might already be doing for ADLS gen2.

[roeap]

@mabasile-MSFT, @jocaplan - thanks for the info, this is great news!

The code in delta-rs is actually just to choose the type of storage backend, we will have to do some (small) udates in the object_store crate where the actual client implementation lives.

However this isn't supported for calls to the ABFS driver, so I'm not sure if that will help you here.

object_store treats all variants or urls (including abfss) as specifying a location, but does not consider a protocol - we always connect using the rest APIs.

For the convenience of the user, we also like to allow for shorthand urls. like s3://<bucket>/<path>, or az://<container>/<path> - in case of blob requiring an additional account_name to be passed via config.

Do you think it makes sense to maybe create a convention for OneLake as well? Maybe something like

onelake://<workspace>/<item>/<path>, assuming that we would always find delta tables in a .lakehouse kind of item?

Also, is there some conventions / guarantees around the top level folders - i.e. would Files map to blobs, and Tables to data stored via the table APIs in storage accounts (and thus not relevant for delta?

Last but not least, we could allow for an optional region parameter to be passed in the config, to control data residency.

I'm on the OneLake team.

May is ask one thing that you may be able to answer or maybe forward to the storage PG? Since listing the storage is a key concern in delta, there are some optimisations possible if the storage supports sorted results and offsets as S3 and GCS do? IIRC, Azure does not support that thus far. Spark and soon delta-rs leverages these optimizations whenever possible. Since it seems MSFT went "all in" on Delta, is there a chance for this to become a feature? :)

[jocaplan]

Happy to pull in the storage team and discuss in more detail. It might be easier to have a quick call if you are open to it. I can send an invite. There are certainly a lot of optimizations that you could do for OneLake since we have a well-known structure at the top and that our tables are in delta lake.

However, I am curious why the normal ADLS path doesn't work. Other than the domain in the URL, we did a lot of work to match the ADLS DFS/Blob endpoints so that existing ADLS apps just work even if they call the REST APIs directly. We mapped our top-level concepts to ADLS concepts like storage accounts and containers. So far, any apps that didn't work were because of 1 of 2 reasons. Either we had a bug on our side, or the app was validating the domain in the URL.

[djouallah]

any update ?

[mabasile-MSFT]

We should have a PR out this week with our fix!

[roeap]

@jocaplan @mabasile-MSFT - sorry for being MIA for a while.

We should have a PR out this week with our fix!

Does this mean you are adding onelake support to object store, or is there another way in the works?

Regarding the talk with the storage team .- if that is still on the table... I am right now working on the ruist delta kernel implentation, trying to apply what we learned during delta-rs. As it turs nout there seems to be an assumtion in spark, the one gets lexicographically sorted results from list, and it seems to work out. But Icould not find any reference in the docs.

A second question would be if there is any change we can one day pass in a blobs key rather then an opaque marker to have list with offset.

Any info on that would ne great :)

[mabasile-MSFT]

Yes, we have two draft PRs - one adding OneLake support to object store, and one adding support in Delta RS (by updating the URL validation). We're just performing the last checks on our side and then we'll have the PRs officially submitted!