wip - CometNativeScan #1076
Merged
wip - CometNativeScan #1076
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
viirya
reviewed
Nov 11, 2024
Comment on lines
80
to
87
| val COMET_FULL_NATIVE_SCAN_ENABLED: ConfigEntry[Boolean] = conf( | ||
| "spark.comet.native.scan.enabled") | ||
| .doc( | ||
| "Whether to enable the fully native scan. When this is turned on, Spark will use Comet to " + | ||
| "read supported data sources (currently only Parquet is supported natively)." + | ||
| " By default, this config is true.") | ||
| .booleanConf | ||
| .createWithDefault(true) |
There was a problem hiding this comment.
I added a similar config locally. 🙂
There was a problem hiding this comment.
If it was was something related, you can merge it here. This is an experimental branch. If it is for something different, then we can change our name.
There was a problem hiding this comment.
No, I meant I tried to do similar native scan change into this branch. I was trying to enable DataFusion native Parquet scan yesterday. I found this branch this morning so tried to apply initial change onto this branch.
I will see if any difference can be merged.
andygrove
reviewed
Nov 12, 2024
| _, | ||
| _, | ||
| _) | ||
| if CometNativeScanExec.isSchemaSupported(requiredSchema) |
There was a problem hiding this comment.
we could also fall back for bucketed scans and other unsupported cases (can be a separate PR)
andygrove
added a commit
that referenced
this pull request
Nov 12, 2024
This reverts commit 38e32f7.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Replaces hacked
CometScanwithCometNativeScanso we can switch between original and experimental without rebuilding.I tried removing a lot of the functions in CometNativeScan but too many test in CometExecSuite fail and I ended up including almost everything back.
Three tests in CometExecScan are still failing.