Reading delta table as pyarrow dataset does not work

[dudzicp]

Describe the bug
I am unable to display the contents of delta tables stored locally

To Reproduce

[tool.poetry.dependencies]
python = "^3.10"
datafusion = "^0.7.0"
deltalake = "^0.6.4"

then run the following code:

import pyarrow as pa
import pyarrow.dataset as ds

from deltalake import DeltaTable
import datafusion

ctx = datafusion.SessionContext()

delta_table = DeltaTable("/local_delta_path/")
pa_dataset = dt.to_pyarrow_dataset()

ctx.register_dataset("pa_dataset", pa_dataset)

tmp = ctx.sql("SELECT * FROM pa_dataset limit 10")
tmp.show()

When executed in notebook in vs code, this script can run for >20 min and I am unable to interrupt the execution.

Expected behavior
Top rows displayed

[dudzicp]

I have also tried to convert table to dataset in the following way:

dataset = dt.to_pyarrow_dataset(
    parquet_read_options=ParquetReadOptions(coerce_int96_timestamp_unit="ms")
)

but the result is the same

[kdbrooks]

@dudzicp - Could you give us the delta files to reproduce this?

[jordandakota]

Where in the code snippet is dt defined, or did you mean delta_table

[dudzicp]

Yes, I meant delta_table. It's been 5 months since I have reported this. Let me reproduce this with latest version of datafusion so then I will provide sample delta table.

[timsaucer]

I have been able to read delta tables using the approach above on DF 42.0.0. However it can be extremely slow for large tables because we don't push down filters. A follow up solution will be provided in #921 and a corresponding update to delta-rs.

I am going to close this because the immediate problem is resolved. Please reopen the issue if you continue to have problems.