Tracking for Iceberg cloud enablement

[nameexhaustion]

Roadmap for allowing distributed iceberg scans in Polars Cloud.

MVP

• Support native dispatch to scan_parquet() with transparent fallback
  • refactor: Add structure for dispatching iceberg to native scans #22405
  • Users can force fallback scans by passing reader_override='pyiceberg'
• Support schema-evolved datasets via type-casting in multi-scan post-apply pipeline
  • Expose cast options parameter to scan_parquet
    • feat(python): Add cast_options parameter to control type casting in scan_parquet #22617
  • Expose extra_columns parameter to scan_parquet
    • feat(python): Add extra_columns parameter to scan_parquet #22699
• Native support for deletion files in multi-scan post-apply pipeline
  • (1) refactor: Propagate iceberg position delete information to IR #23045
  • (2) refactor: Add components for Iceberg deletion file support #23059
  • (3) feat: Native implementation for Iceberg positional deletes #23091
• Set appropriate cast_options / extra_columns etc. parameters when calling native scan_parquet()
  • feat: Enable default set of ScanCastOptions for native scan_iceberg() #23416
  • (Related) Enable use of ScanCastOptions in Delta scans by default (ES) feat: Support reading nanosecond/Int96 timestamps and schema evolved datasets in scan_delta() #23398
• Parquet row-group skipping with type-casting
  • feat: Support row group skipping with filters when cast_options is given #23356
• Column-mapping support Incorrect results on native Iceberg scans when columns have been renamed #23428
  • refactor(rust): Add logic to support struct field renames on arbitrary nesting levels #23532
  • refactor: Propagate Iceberg physical ID schema to IR #23671
  • fix: Fix incorrect native Iceberg scan from tables with renamed/dropped columns/fields #23713
• Parquet row-group skipping with column-mapping
  • feat: Pre-filtered decode and row group skipping with Iceberg / Delta / scans with cast options #23792

After completing the above, we should be safe to switch scan_iceberg() to use the native Parquet scanner by default.

Further work

• Filtering on Iceberg statistics
• Filtering on Iceberg partitions fields
• Fast-count (physical and deleted row counts are available in the Iceberg python objects)
• Parquet pre-filtering with type-casting
  • feat: Pre-filtered decode and row group skipping with Iceberg / Delta / scans with cast options #23792
• Parquet pre-filtering with deletion files
• Parquet pre-filtering with column mapping
  • feat: Pre-filtered decode and row group skipping with Iceberg / Delta / scans with cast options #23792

(ES) - Related to enterprise support work