Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Spark] Support OPTIMIZE tbl FULL for clustered table #3793

Merged
merged 10 commits into from
Oct 30, 2024
Merged

[Spark] Support OPTIMIZE tbl FULL for clustered table #3793

Changes from 1 commit
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
10fa537
add sql word
dabao521 Oct 21, 2024
15bca66
fix up
dabao521 Oct 21, 2024
0ad00a6
add error class to capture wrong FULL use
dabao521 Oct 21, 2024
767715b
fix up
dabao521 Oct 21, 2024
ca2330a
fix up isFull into optimize command context
dabao521 Oct 22, 2024
a664345
Implementation details for OPTIMIZE FULL
dabao521 Oct 22, 2024
92ef939
fix test
dabao521 Oct 23, 2024
7e9a031
fix test failures
dabao521 Oct 23, 2024
0264863
add comment
dabao521 Oct 23, 2024
6ae768a
fix comments
dabao521 Oct 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix test failures
  • Loading branch information
@dabao521
dabao521 committed Oct 23, 2024
commit 7e9a031099745a76926703d48cd2ce992f431737
2 changes: 0 additions & 2 deletions spark/src/main/scala/org/apache/spark/sql/delta/commands/OptimizeTableStrategy.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -252,7 +252,6 @@ case class ClusteringStrategy(
// Keep all files if isFull is set, otherwise skip files with different clusteringProviders
// or files clustered by a different set of clustering columns.
val (candidateFiles, skippedClusteredFiles) = files.iterator.map { f =>
val zCubeInfo = ZCubeInfo.getForFile(f)
clusteringStatsCollector.inputStats.updateStats(f)
f
}.partition { file =>
Expand All @@ -275,7 +274,6 @@ case class ClusteringStrategy(
val finalFiles = (smallZCubeFiles.map(_.addFile) ++ skippedClusteredFiles).toSeq
finalFiles.map { f =>
clusteringStatsCollector.outputStats.updateStats(f)
val zCubeInfo = ZCubeInfo.getForFile(f)
f
}
} else {
Expand Down
Loading