Long-running Spark rewrite Files Action may lead to OutOfMemoryError

[Zhanxiao-Ma]

In my production environment, I have observed that long-running Spark rewrite Files Action can lead to OutOfMemoryError. Analyze the Java dump, I noticed a large number of ChildAllocator objects that are only referenced by the RootAllocator. Upon reviewing the code, I discovered that the ChildAllocator allocated at this point is indeed not being released. Is this correct?

iceberg/arrow/src/main/java/org/apache/iceberg/arrow/vectorized/VectorizedReaderBuilder.java

Line 59 in cbb8530

this.rootAllocator =

[Zhanxiao-Ma]

I think the close method of VectorizedArrowReader should include the logic to release rootAlloc.

[RussellSpitzer]

Do you have any other details on this? Possibly a way to test this deterministically? I think if we had more information about the length of the job, the amount of work, and other details we could probably determine if this is a problem we need to fix and take further steps.