Use Iceberg-Rust for parsing the ManifestList and Manifests

[Fokko]

Rationale for this change

This replaces the Cython implementation for reading Avro with Iceberg-Rust. This would greatly simplify the PyIceberg project since we don't have to publish Python wheels anymore.

Are these changes tested?

Are there any user-facing changes?

[Fokko]

@roeap Thanks, your patch fixed correctly passing through the 102: partition field 🥳

[kevinjqliu]

looks like we're waiting for apache/iceberg-rust#1328 to finalize, which is dependent on apache/iceberg-rust#1482 :)

[yogevyuval]

@Fokko This is awesome! thanks for your work on this one.
Do we expect any performance/memory differences with reading it with the rust module?

[Fokko]

@yogevyuval Thanks for asking, and yes, I do expect performance impact since just a part of the deserialization is cythonized. With this change, much more is pushed into Rust. This will also reduce the GIL pressure since we don't have to build the readers anymore :)

I left this PR small to focus on the essentials, but once this is in, we can also clean up a LOT of code 👍

[yogevyuval]

@yogevyuval Thanks for asking, and yes, I do expect performance impact since just a part of the deserialization is cythonized. With this change, much more is pushed into Rust. This will also reduce the GIL pressure since we don't have to build the readers anymore :)

I left this PR small to focus on the essentials, but once this is in, we can also clean up a LOT of code 👍

Interesting, we have quite a bit of concurrency going on so curious to see the results. Once apache/iceberg-rust#1328 gets merged and release i'll try and create some basic benchmarks and share the results

[Fokko]

@yogevyuval That's not good news, but thanks for checking 👍 I agree that we should find out what's going on before merging this. Thanks for doing some testing, that's very much appreciated