Skip to content

[AURON #2581] Extract supported conjuncts for Iceberg changelog task … - #2583

Open
Sigma-Ma wants to merge 1 commit into
apache:masterfrom
Sigma-Ma:Auron-2581-extract-supported-conjuncts-for-iceberg-changelog-task-pruning
Open

Sigma-Ma wants to merge 1 commit into
apache:masterfrom
Sigma-Ma:Auron-2581-extract-supported-conjuncts-for-iceberg-changelog-task-pruning

Conversation

@Sigma-Ma

@Sigma-Ma Sigma-Ma commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #2581

Rationale for this change

A metadata predicate combined with an ordinary data-column predicate currently prevents Iceberg changelog task pruning. For example, _change_type = 'INSERT' AND id = 3 retains DELETE tasks even though their metadata excludes them.

What changes are included in this PR?

  • Extract supported top-level AND conjuncts and validate each conjunct against the scan output by expression ID.
  • Reuse the existing changelog task filter and preserve the complete post-scan condition.
  • Keep OR/NOT boundaries, operator barriers, nondeterministic filters, task eligibility, and runtime-filter propagation conservative.
  • Extend regression coverage for mixed INSERT/DELETE ranges, nested AND conditions, projected data attributes, expression-ID mismatches, empty results, and runtime-filter copies. Assert expected rows, retained task counts, and file metrics independently.

Are there any user-facing changes?

Eligible native Iceberg changelog scans can read fewer tasks when metadata predicates are combined with data-column conditions. No public API or configuration changes.

How was this patch tested?

Build dependencies:

./build/mvn -o -ntp -Ppre,spark-3.5,scala-2.12,iceberg-1.10.1 -pl thirdparty/auron-iceberg -DskipBuildNative -DwildcardSuites=org.apache.auron.iceberg.AuronIcebergIntegrationSuite spotless:apply test

./dev/reformat --check

Was this patch authored or co-authored using generative AI tooling?

  • Yes
  • No

Generated-by: OpenAI Codex (GPT-6)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extract supported conjuncts for Iceberg changelog task pruning

2 participants