Skip to content

[BLAZE-707][FOLLOWUP] NativePaimonTableScanExec should use shimed PartitionedFile and min partition number#713

Merged
richox merged 1 commit intoapache:masterfrom
SteNicholas:BLAZE-707
Dec 21, 2024
Merged

[BLAZE-707][FOLLOWUP] NativePaimonTableScanExec should use shimed PartitionedFile and min partition number#713
richox merged 1 commit intoapache:masterfrom
SteNicholas:BLAZE-707

Conversation

@SteNicholas
Copy link
Member

Which issue does this PR close?

Closes #707.

Rationale for this change

PartitionedFile has different constructors among different Spark versions. Meanwhile, the min partition number of max splits is also different for different Spark versions. NativePaimonTableScanExec should use shimed PartitionedFile and min partition number for compilation error as follows:

  • spark-3.0
Error: ] /home/runner/work/blaze/blaze/spark-extension/src/main/scala/org/apache/spark/sql/hive/execution/blaze/plan/NativePaimonTableScanExec.scala:232: value filesMinPartitionNum is not a member of org.apache.spark.sql.internal.SQLConf

  • spark-3.4, spark-3.5
Error: ] /home/runner/work/blaze/blaze/spark-extension/src/main/scala/org/apache/spark/sql/hive/execution/blaze/plan/NativePaimonTableScanExec.scala:219: type mismatch;
 found   : String
 required: org.apache.spark.paths.SparkPath
Error: ] /home/runner/work/blaze/blaze/spark-extension/src/main/scala/org/apache/spark/sql/hive/execution/blaze/plan/NativePaimonTableScanExec.scala:222: type mismatch;
 found   : String
 required: org.apache.spark.paths.SparkPath

What changes are included in this PR?

NativePaimonTableScanExec uses shimed PartitionedFile and min partition number to fix compilation error.

Are there any user-facing changes?

No.

@richox
Copy link
Contributor

richox commented Dec 21, 2024

seems ci is still broken, i will fix it later.

@richox richox merged commit 43e3621 into apache:master Dec 21, 2024
@richox richox mentioned this pull request Apr 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support native scan paimon cow table

2 participants