Skip to content

Extract supported conjuncts for Iceberg changelog task pruning #2581

Open

Description

@Sigma-Ma

Is your feature request related to a problem? Please describe.

Auron currently accepts a changelog task filter only when the entire filter expression is supported. For AND, both operands must satisfy isSupportedChangelogTaskFilter.

As a result, adding an ordinary data-column predicate prevents an otherwise supported metadata predicate from being used for task pruning. For example, on a directly filtered changelog scan:

SELECT id, amount, _change_type
FROM catalog.db.t.changes
WHERE _change_type = 'INSERT' AND amount > 100;

The _change_type condition could exclude DELETE tasks, but the combined condition is rejected by this pruning path. Results remain correct because the complete filter is still evaluated after scanning.

The existing direct-scan test explicitly retains all three tasks for a condition combining _commit_snapshot_id with an ordinary data column:

https://github.com/apache/auron/blob/master/thirdparty/auron-iceberg/src/test/scala/org/apache/auron/iceberg/AuronIcebergIntegrationSuite.scala#L695

Describe the solution you'd like

Extract supported metadata conjuncts from the top-level AND chain and use them for changelog task pruning, while retaining the complete post-scan filter.

Reuse the existing predicate support checks and task-level metadata evaluator for _change_type, _commit_snapshot_id, and _change_ordinal.

Keep the initial scope conservative:

  • Split only AND; do not extract predicates from inside OR or NOT.
  • Validate each extracted conjunct against the scan output by expression ID.
  • Preserve the current operator boundaries and deterministic-projection checks.
  • Preserve runtime-filter tag propagation and fallback for unsupported scan tasks.
  • Apply pruning to the INSERT and full-data-file DELETE tasks already supported by the native scan.
  • Add no configuration or public API.

Validation should cover:

  • Mixed metadata/data predicates, with data-column filtering removing some rows from retained tasks.
  • Multiple metadata conjuncts combined with ordinary data predicates, including nested AND expressions.
  • Mixed INSERT/full-data-file DELETE ranges and predicates excluding every task.
  • OR/NOT boundaries, expression-ID mismatches, unsafe operators, and runtime-filter scan copies.

Compare explicit expected rows with Spark execution and assert retained native scan task counts. Use controlled file layouts for file-metric assertions; result equality alone does not demonstrate pruning.

Describe alternatives you've considered

Continue evaluating the complete condition after scanning. This preserves results but retains tasks that supported metadata conjuncts can already exclude.

Additional context

This extends the metadata pruning introduced in #2456 and the full-data-file DELETE task pruning added in #2523. Mixed metadata/data predicates were outside the original scope.

The expected implementation is a focused change to IcebergConvertProvider.prepare with regression coverage in AuronIcebergIntegrationSuite, reusing IcebergScanSupport.

The optimization reduces retained native scan tasks after Iceberg input-task planning. It does not broaden pruning across changelog-view processing or other semantic boundaries.

This proposal is based on source inspection and existing test assertions. No new reproduction or performance benchmark has been run.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions