Data Integrity Issue with DELETE Operation Using Copy-on-Write (COW) and Equality Deletes

[and124578963]

Apache Iceberg version

1.8.1 (latest release)

Query engine

Spark

Please describe the bug 🐞

Description

When executing a sequence of deletes (position deletes followed by equality deletes and a final row deletion) in Copy-on-Write (COW) mode, the equality deletes are not applied to the original data files, resulting in residual data that should have been removed.

Observed Context:

• The issue does not occur when combining UPDATE and MERGE operations in COW mode – these work as expected.
• The problem is specific to COW; Merge-on-Read (MOR) mode handles the same scenario correctly.

Steps to Reproduce

1) Data Setup:

• Create two Parquet data files:

  • data-file-1.parquet: IDs [1, 2, 3, 4, 5]

  • data-file-2.parquet: IDs [6, 7, 8, 9, 10]

• Configure the table with COW semantics (write.delete.mode = copy-on-write).

2) Apply Initial Deletes:

• Add a position delete file to remove:

  • Row 0 (ID 1) from data-file-1.parquet

  • Row 0 (ID 6) from data-file-2.parquet

• Add an equality delete file targeting IDs [3, 4, 5, 6, 7, 8, 9, 10].

3) Execute Final Delete Command:

DELETE FROM table WHERE id = 2; -- Targets remaining ID '2'

Expected Result

After all deletions:

• SELECT * FROM table should return no rows, as:

  • Position deletes remove IDs 1 and 6.

  • Equality deletes remove IDs 3, 4, 5 (from data-file-1) and 7, 8, 9, 10 (from data-file-2).

  • Final DELETE WHERE id = 2 removes the last remaining ID (2).

Actual Result

SELECT * FROM table returns IDs 3, 4, 5.

• Observed Issues:

The equality deletes targeting 3, 4, 5 (in data-file-1) are not applied.

The DELETE WHERE id = 2 operation only removes ID 2, leaving 3, 4, 5 intact.

Environment

Apache Iceberg Versions: 1.6.1, 1.8.1

Tests

Example of tests:
1.8.x...and124578963:iceberg:1.8.x
To run:
./gradlew :iceberg-spark:iceberg-spark-3.5_2.12:test --tests "org.apache.iceberg.spark.TestSparkExecutionWithEqualityAndPositionDeletes"

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[RussellSpitzer]

Checking this out in the debugger, the Scan produced by CopyOnWrite is not including the equality delete in it's Row level Operation. Will update when I find out why.

[RussellSpitzer]

Figured it out.

The equality delete files are eliminated from scan planning because we assume that the filters exclude any matching rows from being evaluated. This is incorrect because in COW mode we do need all rows to be evaluated even if they are not part of the current operation.

Example

Table has an equality delete removing rows where X = 3
We issue a command "DELETE WHERE X = 2"
When planning we currently incorrectly assume we don't need to look at the equality delete since we aren't touching rows where X = 3
The file is read, rows where X = 3 are materialized and copied into the new data file.

CC @aokolnychyi

[sfc-gh-ygu]

It should not be allowed to apply pos deletes and eq deletes when write.delete.mode = copy-on-write. NO?

[RussellSpitzer]

@sfc-gh-ygu

It has to apply both in Copy on Write.

So imagine I have a Data File

data.parquet

X
1
2
3
4

And equality Delete
eq.parquet

x
3

When I do a scan of this file without a filter I make a scan task

{ 
  file = data.parquet
  deleteList = eq.parquet
}

The problem comes when I apply a filter,

So say I do DELETE WHERE x = 2

This produces a scan with a filter pushdown of x = 2 which is used in table.scan.filter

The filter condition is then checked against eq.parquet which has a min and max for x of 3. Since we know x = 2 we get a "CAN NOT MATCH" and ignore eq.delete.

So I produce a scan task that looks like

{ 
  file = data.parquet
  deleteList = nil
}

This scan task goes through the COW execution path which performs DELETE WHERE x = 2 to every row but since we are in COW rows that are not deleted are shunted into a new file. Here we have a problem because we aren't applying the equality deletes so we write a new file

data_2.parquet

X
1
3
4

[sfc-gh-ygu]

It's fine if the eq delete files will still be picked up at next read. But I guess we did filter it out based on the sequence number, so that the previous eq delete files won't be there at read time.

  private Iterable<DeleteFile> filterDeleteFiles() {
      return Iterables.filter(deleteFiles, file -> file.dataSequenceNumber() > minSequenceNumber);
    }

[RussellSpitzer]

Wrote a smaller reproduction here
https://github.com/apache/iceberg/pull/12479/files

Working on a fix

Note the above repo passes for SparkCatalog because that test is using "avro" file format and this avoids using metadata for pruning

[RussellSpitzer]

Getting closer, the problem is that "ignoreResiduals" is actually being ignored (set to false) when we actually perform the read. Trying to figure out why.

[RussellSpitzer]

https://github.com/apache/iceberg/pull/12479/files Fix posted