Repository navigation
Extract supported conjuncts for Iceberg changelog task pruning #2581
Description
Is your feature request related to a problem? Please describe.
Auron currently accepts a changelog task filter only when the entire filter expression is supported. For AND, both operands must satisfy isSupportedChangelogTaskFilter.
As a result, adding an ordinary data-column predicate prevents an otherwise supported metadata predicate from being used for task pruning. For example, on a directly filtered changelog scan:
SELECT id, amount, _change_type
FROM catalog.db.t.changes
WHERE _change_type = 'INSERT' AND amount > 100;The _change_type condition could exclude DELETE tasks, but the combined condition is rejected by this pruning path. Results remain correct because the complete filter is still evaluated after scanning.
The existing direct-scan test explicitly retains all three tasks for a condition combining _commit_snapshot_id with an ordinary data column:
Describe the solution you'd like
Extract supported metadata conjuncts from the top-level AND chain and use them for changelog task pruning, while retaining the complete post-scan filter.
Reuse the existing predicate support checks and task-level metadata evaluator for _change_type, _commit_snapshot_id, and _change_ordinal.
Keep the initial scope conservative:
- Split only
AND; do not extract predicates from insideORorNOT. - Validate each extracted conjunct against the scan output by expression ID.
- Preserve the current operator boundaries and deterministic-projection checks.
- Preserve runtime-filter tag propagation and fallback for unsupported scan tasks.
- Apply pruning to the INSERT and full-data-file DELETE tasks already supported by the native scan.
- Add no configuration or public API.
Validation should cover:
- Mixed metadata/data predicates, with data-column filtering removing some rows from retained tasks.
- Multiple metadata conjuncts combined with ordinary data predicates, including nested
ANDexpressions. - Mixed INSERT/full-data-file DELETE ranges and predicates excluding every task.
- OR/NOT boundaries, expression-ID mismatches, unsafe operators, and runtime-filter scan copies.
Compare explicit expected rows with Spark execution and assert retained native scan task counts. Use controlled file layouts for file-metric assertions; result equality alone does not demonstrate pruning.
Describe alternatives you've considered
Continue evaluating the complete condition after scanning. This preserves results but retains tasks that supported metadata conjuncts can already exclude.
Additional context
This extends the metadata pruning introduced in #2456 and the full-data-file DELETE task pruning added in #2523. Mixed metadata/data predicates were outside the original scope.
The expected implementation is a focused change to IcebergConvertProvider.prepare with regression coverage in AuronIcebergIntegrationSuite, reusing IcebergScanSupport.
The optimization reduces retained native scan tasks after Iceberg input-task planning. It does not broaden pruning across changelog-view processing or other semantic boundaries.
This proposal is based on source inspection and existing test assertions. No new reproduction or performance benchmark has been run.
Activity