Skip to content

Conversation

@comphead
Copy link
Contributor

@comphead comphead commented Sep 3, 2025

Which issue does this PR close?

Closes #.

Rationale for this change

Comet supports multiple scan implementations (e.g., native Comet, DataFusion, Iceberg-compat) and often runs in SCAN_AUTO mode to pick the best option at runtime. Previously, that selection was recomputed each time the rule ran, which adds some overhead. Moreover it is not easy to know in runtime what value has been picked up.

This PR derives the scan implementation once and stores it at the session level, reusing the same choice for subsequent scans. It preserves existing selection semantics, including falling back to native Comet when Parquet encryption is enabled and otherwise delegating to selectScan.
The selected implementation is now exposed via CometScanRule.getCurrentScanImplementation

What changes are included in this PR?

How are these changes tested?

@andygrove
Copy link
Member

I don't think that this approach can really work since once query could use multiple scan implementations (for example, we may fall back to native_comet for scans against tables using byte/short and use native_iceberg_compat for other scans)

@comphead comphead closed this Sep 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants