Rust writer not encoding correct URL for partitions in delta table

[gprashmi]

Environment

Delta-rs version: 0.17.4

---

Bug

We write data to delta table using delta-rs with PyArrow engine with DayHour as partition column. However when we run the optimize.compact() on the table, it creates partitions with spaces and does not properly encode the partition urls as shown in the below image i.e; it creates new partitions url with spaces (.zstd.parquet).

Can you please let me know how we can run the optimize.compact without having partitions with spaces?

[gprashmi]

@g12-al

[g12-al]

Confirmed that this also seems to happen in 0.18.0.

This breaks compatibility for our Trino connector to enable visualization on dashboards. Ideally, Trino wouldn't care about spaces (it has a lot of other issues like not being compatible with timezone-aware timestamps).

[gprashmi]

@ion-elgreco the issue we see here is with the Pyarrow engine

[gprashmi]

@ion-elgreco on-elgreco Any update on this issue?

[ion-elgreco]

@gprashmi sorry I don't have time to look into it unfortunately

[gprashmi]

@ion-elgreco do you have any other suggestions on how this can be handled to have the correct URL encoding with optimizing the table?

[ion-elgreco]

I'll take a look at this

[ion-elgreco]

@gprashmi @g12-al fix is on it's way in #2705, will be released in 0.18.3