Flaky test: TestSparkExecutorCache > testMergeOnReadDelete()

[ajantha-bhat]

build: https://github.com/apache/iceberg/actions/runs/7569167293/job/20611887361
PR: #9507

position deletes are missing? cache?

TestSparkExecutorCache > testMergeOnReadDelete() > catalogName = testhive, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hive, io-impl=org.apache.iceberg.spark.TestSparkExecutorCache$CustomFileIO, default-namespace=default} FAILED
    java.lang.AssertionError: 
    Expecting all elements of:
      [GenericDeleteFile{content=position_deletes, file_path=/tmp/junit17911827767769921495/pos-deletes-da2fc9e8-2fc7-42ad-901b-9139ee8102b0, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=1267, column_sizes={2147483546=145, 2147483545=69}, value_counts=null, null_value_counts=null, nan_value_counts=null, lower_bounds=null, upper_bounds=null, key_metadata=null, split_offsets=[4], equality_ids=null, sort_order_id=null, data_sequence_number=null, file_sequence_number=null},
        GenericDeleteFile{content=equality_deletes, file_path=/tmp/junit17911827767769921495/eq-deletes-15c2d19c-c3dd-4b17-95cc-7e68759550f0, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=448, column_sizes={1=46}, value_counts={1=2}, null_value_counts={1=0}, nan_value_counts={}, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@e1782, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@e1787, key_metadata=null, split_offsets=[4], equality_ids=[1], sort_order_id=0, data_sequence_number=null, file_sequence_number=null}]
    to match given predicate but this element did not:
      GenericDeleteFile{content=equality_deletes, file_path=/tmp/junit17911827767769921495/eq-deletes-15c2d19c-c3dd-4b17-95cc-7e68759550f0, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=448, column_sizes={1=46}, value_counts={1=2}, null_value_counts={1=0}, nan_value_counts={}, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@e1782, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@e1787, key_metadata=null, split_offsets=[4], equality_ids=[1], sort_order_id=0, data_sequence_number=null, file_sequence_number=null}
        at org.apache.iceberg.spark.TestSparkExecutorCache.checkDelete(TestSparkExecutorCache.java:268)
        at org.apache.iceberg.spark.TestSparkExecutorCache.testMergeOnReadDelete(TestSparkExecutorCache.java:254)
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has 

[manuzhang]

The error says the equality delete file doesn't match the predicate, ie streamCount(deleteFile) <= 1.

    int maxRequestCount = mode == COPY_ON_WRITE ? 3 : 1;
    assertThat(deleteFiles).allMatch(deleteFile -> streamCount(deleteFile) <= maxRequestCount);

[aokolnychyi]

I'll look into it, probably related to some cache invalidation.

[manuzhang]

It looks this is happening again https://github.com/apache/iceberg/actions/runs/10486032923/job/29043393618

TestSparkExecutorCache > testMergeOnReadUpdate() > catalogName = testhive, implementation = org.apache.iceberg.spark.SparkCatalog, config = {type=hive, io-impl=org.apache.iceberg.spark.TestSparkExecutorCache$CustomFileIO, default-namespace=default} FAILED
    java.lang.AssertionError: 
    Expecting all elements of:
      [GenericDeleteFile{content=position_deletes, file_path=/tmp/junit17759035431822043198/pos-deletes-bb0e7a65-0665-4aca-8420-4b1d350158fb, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=1267, column_sizes={2147483546=147, 2147483545=69}, value_counts=null, null_value_counts=null, nan_value_counts=null, lower_bounds=null, upper_bounds=null, key_metadata=null, split_offsets=[4], equality_ids=null, sort_order_id=null, data_sequence_number=null, file_sequence_number=null},
        GenericDeleteFile{content=equality_deletes, file_path=/tmp/junit17759035431822043198/eq-deletes-44c2c956-e9c2-4a5f-8d99-8cea6fdfa6a4, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=448, column_sizes={1=46}, value_counts={1=2}, null_value_counts={1=0}, nan_value_counts={}, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@e1782, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@e1787, key_metadata=null, split_offsets=[4], equality_ids=[1], sort_order_id=0, data_sequence_number=null, file_sequence_number=null}]
    to match given predicate but this element did not:
      GenericDeleteFile{content=equality_deletes, file_path=/tmp/junit17759035431822043198/eq-deletes-44c2c956-e9c2-4a5f-8d99-8cea6fdfa6a4, file_format=PARQUET, spec_id=0, partition=PartitionData{}, record_count=2, file_size_in_bytes=448, column_sizes={1=46}, value_counts={1=2}, null_value_counts={1=0}, nan_value_counts={}, lower_bounds=org.apache.iceberg.SerializableByteBufferMap@e1782, upper_bounds=org.apache.iceberg.SerializableByteBufferMap@e1787, key_metadata=null, split_offsets=[4], equality_ids=[1], sort_order_id=0, data_sequence_number=null, file_sequence_number=null}
        at org.apache.iceberg.spark.TestSparkExecutorCache.checkUpdate(TestSparkExecutorCache.java:303)
        at org.apache.iceberg.spark.TestSparkExecutorCache.testMergeOnReadUpdate(TestSparkExecutorCache.java:286)