[Epic] Full Delta Lake Integration #171
Description
https://github.com/delta-io/delta-rs
https://github.com/delta-io/delta-kernel-rs
Milestones
- Milestone 1: Delta table operations in Spark SQL and DataFrame APIs.
- Read/Write operations.
- Basic READ/WRITE
- Partition Write
- Data Skipping
- Schema Handling
- DML Operations
- UPDATE
- DELETE
- MERGE INTO
- Schema Evolution
- Time Travel
- Table Maintenance
- OPTIMIZE
- VACUUM
- TableProperties
- Change Data Feed
- Transaction & Concurrency
- Read/Write operations.
- Milestone 2: Catalog integration.
- Unity Catalog
- AWS Glue
- Milestone 3: Distributed Delta table operations
- Milestone 4: Streaming support
Other Milestones to be mentioned later.
Related Work
- feat: basic read/write operations for Delta Lake #578
- feat: partitioned writes and data skipping for Delta Lake #634
- refactor: pass url directly for Delta Lake #677
- feat: schema evolution for Delta Lake #680
- docs: add guide for Delta Lake #717
- feat: support Delta codec #723
- fix: remove DeltaScan and unify schema evolution paths for Delta Lake #750
- feat: support DeltaPlan for Delta Lake #768
- refactor: kernel log replay for Delta Lake #794
- fix: writing Delta table with partitioned inputs #837
- refactor: handle side-effect of replaceWhere for Delta Lake #848
- feat: time travel for Delta Lake #854
- feat: delete operation for Delta Lake #856
- fix: handle side-effect of delete operation for Delta Lake #916
- refactor: share scan utilities for Delta Lake #921