[EPIC] Faster manifest reads through pyiceberg-core #4007

Feature Request / Improvement

PyIceberg reads manifests and manifest lists with its Cython Avro decoder. pyiceberg-core (the iceberg-rust Python binding) already exposes read_manifest_entries and read_manifest_list, but using them today makes manifest reads 4x to 5x slower than Cython. This issue tracks the work in both repos to make the pyiceberg-core path faster than Cython, and then to switch PyIceberg to it. The iceberg-rust side is tracked in apache/iceberg-rust#3262, which repeats those tasks in more detail.

The earlier attempt (apache/iceberg-rust#1280, #2493) was closed. It converted every Rust manifest entry into a PyIceberg DataFile in Python, which is the rust_convert path measured below. The full benchmark writeup from @kevinjqliu, including an in-crate profile of the Rust parser and the script, is in kevinjqliu#45.

Where the time goes

These numbers come from manifests written by PyIceberg (V2, deflate, 12 columns with full column stats, identity partition). pyiceberg is at main (0d58407) with the Cython decoder. pyiceberg-core is built locally from the v0.10.1 tag, once as released (opt-level = "z") and once with opt-level = 3. Each number is the best of 7 runs in ms, single threaded, on Apple Silicon with Python 3.12. v0.10.1 predates apache/iceberg-rust#3028, which its PR measured as making the Rust parse about 5% faster.

The pyiceberg-core columns include building PyIceberg ManifestEntry and DataFile objects, so all three columns produce the same result. At 10,000 entries with opt-level = 3, the Rust parse costs 21.7 us per entry and converting the result into PyIceberg objects costs another 10.3 us per entry. The whole Cython path costs 10.3 us per entry in the same run. The conversion alone costs as much as Cython, so a faster Rust parser can't reach parity unless PyIceberg also stops building one Python object per entry from the binding's output. apache/iceberg-rust#3354 has since cut the Rust parse to 6.7 us per entry on these manifests, and to 3.7 us per entry on the same entries written with the 64 KB blocks of #4008. The conversion is now the larger cost in both cases. apache/iceberg-rust#3262 now decodes manifests with arrow-avro, which reads the same entries at 4.0 and 1.1 us per entry straight into Arrow.

Manifest lists are closer. With opt-level = 3 the Rust parse of a 10,000-entry manifest list takes 8.0 ms against 23.9 ms for Cython, and the conversion into ManifestFile objects is what makes the end-to-end path slower (39.1 ms).

Task list

These tasks are in PyIceberg.

• Write equality_ids as list<int> per the spec. PR: Manifest: Use int for equality_ids in manifest schema per Iceberg spec (#3840) #3842.
• Flush manifest entries in size-bounded Avro blocks. ManifestWriter.add_entry calls write_block once per entry (manifest.py#L1213-L1238), so every entry gets its own deflate stream and sync marker. Java writes manifests through Avro's DataFileWriter and keeps its default sync interval, which the linked issue reports as 64 KB. In the linked benchmark, a 10,000-entry manifest shrank from 3.6 MB to 0.7 MB and read about 1.5x faster with both decoders after re-encoding with 64 KB blocks. The manifest-list writer already writes one block (manifest.py#L1389). This task doesn't depend on anything else. PR: perf(avro): buffer records into size-bounded blocks #4008
• Build inspect.entries() and inspect.files() from pyiceberg-core Arrow output. Both methods build a DataFile per entry and then flatten them back into a pa.Table (inspect.py#L152, inspect.py#L860). These are the first consumer of the Arrow output from the iceberg-rust tasks, because they want Arrow anyway. This task depends on that output being in a pyiceberg-core release.
• Plan scans through pyiceberg-core. _open_manifest (table/__init__.py#L2321-L2334) filters each entry with Python partition and metrics evaluators while decoding (feat: Fuse partition and metrics filtering into manifest deserialization #3658), but every entry still becomes a DataFile before it is filtered. To beat Cython, filtering has to happen on the Arrow columns, so that only the matching entries become Python objects. iceberg-rust's evaluators take a parsed DataFile, so using them would bring back the slower decode. Whether the filtering runs vectorized in PyIceberg or in an Arrow-based evaluator in Rust is open and is settled together with the iceberg-rust task for filtering. pyiceberg-core is an optional extra, so the Cython reader stays as the fallback.
• Read manifest lists through pyiceberg-core. This uses the same approach as scan planning, applied to read_manifest_list.

These tasks are in iceberg-rust and tracked in apache/iceberg-rust#3262.

• Faster manifest parsing. perf(manifest): upgrade to apache-avro 0.22 and read manifest entries without Value iceberg-rust#3354 upgraded to apache-avro 0.22 and decodes manifest entries directly from the writer schema instead of through apache_avro::Value and schema resolution. In its benchmark at opt-level = 3, parsing a 10,000-entry PyIceberg manifest went from 211.5 ms to 36.8 ms.
• Build the pyiceberg-core wheel with opt-level = 3. In the numbers above, this makes the parse about 1.6x faster and the end-to-end path about 1.4x faster. It also nearly doubles the wheel size.
• Decode manifests and manifest lists into Arrow with arrow-avro and return them from pyiceberg-core. The reader applies the spec's inheritance of snapshot IDs, sequence numbers, and first row IDs from the manifest list, which the Cython reader does today in _inherit_from_manifest (manifest.py#L1050).
• Release pyiceberg-core with the above. PyIceberg pins pyiceberg-core>=0.10.1,<0.11.0.