Pyiceberg leaks memory on table write #1809
Description
Apache Iceberg version
0.9.0 (latest release)
Please describe the bug 🐞
Hi Everyone!
I've encountered memory leak while using pyiceberg on Ubuntu aarch64 slim-container. There is no aarch64 wheel available so pyiceberg/avro/decoder_fast.pyi is not built thus pyiceberg is reverting to native implementation. I'm getting the following warning.
/app/.venv/lib/python3.12/site-packages/pyiceberg/avro/decoder.py:185: UserWarning: Falling back to pure Python Avro decoder, missing Cython implementation
warnings.warn("Falling back to pure Python Avro decoder, missing Cython implementation")
First, I thought StreamingBinaryDecoder (fallback for decoder_fast) will be the culprit but the issue persits in amd64 image where decoder_fast is compiled.
I'm writing pyarrow tables to AWS Glue repeatedly and it crashes with OOM after a couple of writes in the container. Each write leaves ~200mb residue in memory after deletion of the Iceberg Table object.
It works on macOS fine, storage is getting freed.
iceberg_table = self.glue_catalog.load_table(
(self.database, self.table_name)
)
iceberg_table.append(pa_table)
del iceberg_tableTested in image: ghcr.io/astral-sh/uv:python3.12-bookworm-slim
Test envs: M1 macBook natively, M1 macBook Podman amd64 emulation, AWS t3.small Podman
Built: Podman on M1macBook
I'm trying to find out the reason for the leak.
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity
[-]StreamingBinaryDecoder leaks memory in t[/-][+]StreamingBinaryDecoder leaks memory when decoder_fast.pyi is not compiled.[/+][-]StreamingBinaryDecoder leaks memory when decoder_fast.pyi is not compiled.[/-][+]StreamingBinaryDecoder leaks memory when decoder_fast.pyi is not compiled[/+][-]StreamingBinaryDecoder leaks memory when decoder_fast.pyi is not compiled[/-][+]Pyiceberg leaks memory[/+][-]Pyiceberg leaks memory[/-][+]Pyiceberg leaks memory on table write[/+]gabor-one commentedon Mar 19, 2025
I've traced back the issue to PyArrow. Sorry, my mistake.
MoIMC commentedon Apr 10, 2025
Currently experiencing something similar here - how were you able to isolate this to PyArrow @gabor-one ? Thank you!
gabor-one commentedon Apr 14, 2025
@MoIMC I have an application that receive streaming data on websockets in one process and in the other is parses it directly into RecordBatch every 20k records with
pa.RecordBatch.from_pylist. Later from the RecordBatches Table is being built withpa.Table.from_batches. The system kept using up all the available memory on Linux systems (2g+) even when metrics frompa.default_memory_pool()were showing no memory usage by Pyarrow. I suspect that some object were not released in Python when RecordBatch was created from list of dicts, but did not took the time to debug properly.If i remember correctly simply commenting out
iceberg_table.append(pyarrow_table)did not helped the issue, hence the closed ticket. Moved to Google BigQuery as it has streaming ingestion endpoint, and i don't have to beg for getting started credits.dsully commentedon Sep 17, 2025
I'm seeing this as well - is there a work-around?