[SPARK-32056][SQL][Follow-up] Coalesce partitions for repartiotion hint and sql when AQE is enabled #28952
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I also think this might be worth creating a new jira ticket, but as initially we discussed it as follow-up. So I put it as a follow-up first. |
|
ok to test |
|
Does jenkins not work? |
|
Yea, it seems jenkins got sick last night... |
cloud-fan
reviewed
Jun 30, 2020
| case s: ShuffleExchangeExec => s | ||
| } | ||
| assert(shuffle.size == 1) | ||
| assert(shuffle(0).outputPartitioning.numPartitions == 10) |
There was a problem hiding this comment.
let's put 10 as a parameter, to make this method a bit more general.
cloud-fan
approved these changes
Jun 30, 2020
|
+1 looks good to me too |
|
Test build #124649 has finished for PR 28952 at commit
|
|
retest this please |
|
Test build #124684 has finished for PR 28952 at commit
|
|
Test build #124712 has finished for PR 28952 at commit
|
|
retest this please |
|
Test build #124726 has started for PR 28952 at commit |
|
retest this please |
|
Test build #124774 has finished for PR 28952 at commit
|
|
retest this please... |
|
Test build #124804 has finished for PR 28952 at commit
|
dongjoon-hyun
approved these changes
Jul 1, 2020
There was a problem hiding this comment.
+1, LGTM. Thank you, @viirya and @cloud-fan .
Merged to master for Apache Spark 3.1.
|
Thanks all! |
MGHawes
pushed a commit
to palantir/spark
that referenced
this pull request
May 16, 2021
…nt and sql when AQE is enabled As the followup of apache#28900, this patch extends coalescing partitions to repartitioning using hints and SQL syntax without specifying number of partitions, when AQE is enabled. When repartitionning using hints and SQL syntax, we should follow the shuffling behavior of repartition by expression/range to coalesce partitions when AQE is enabled. Yes. After this change, if users don't specify the number of partitions when repartitioning using `REPARTITION`/`REPARTITION_BY_RANGE` hint or `DISTRIBUTE BY`/`CLUSTER BY`, AQE will coalesce partitions. Unit tests. Closes apache#28952 from viirya/SPARK-32056-sql. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
LorenzoMartini
pushed a commit
to palantir/spark
that referenced
this pull request
May 18, 2021
…nt and sql when AQE is enabled As the followup of apache#28900, this patch extends coalescing partitions to repartitioning using hints and SQL syntax without specifying number of partitions, when AQE is enabled. When repartitionning using hints and SQL syntax, we should follow the shuffling behavior of repartition by expression/range to coalesce partitions when AQE is enabled. Yes. After this change, if users don't specify the number of partitions when repartitioning using `REPARTITION`/`REPARTITION_BY_RANGE` hint or `DISTRIBUTE BY`/`CLUSTER BY`, AQE will coalesce partitions. Unit tests. Closes apache#28952 from viirya/SPARK-32056-sql. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
LorenzoMartini
pushed a commit
to palantir/spark
that referenced
this pull request
May 19, 2021
…nt and sql when AQE is enabled As the followup of apache#28900, this patch extends coalescing partitions to repartitioning using hints and SQL syntax without specifying number of partitions, when AQE is enabled. When repartitionning using hints and SQL syntax, we should follow the shuffling behavior of repartition by expression/range to coalesce partitions when AQE is enabled. Yes. After this change, if users don't specify the number of partitions when repartitioning using `REPARTITION`/`REPARTITION_BY_RANGE` hint or `DISTRIBUTE BY`/`CLUSTER BY`, AQE will coalesce partitions. Unit tests. Closes apache#28952 from viirya/SPARK-32056-sql. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
16pierre
pushed a commit
to 16pierre/spark
that referenced
this pull request
May 24, 2021
…nt and sql when AQE is enabled As the followup of apache#28900, this patch extends coalescing partitions to repartitioning using hints and SQL syntax without specifying number of partitions, when AQE is enabled. When repartitionning using hints and SQL syntax, we should follow the shuffling behavior of repartition by expression/range to coalesce partitions when AQE is enabled. Yes. After this change, if users don't specify the number of partitions when repartitioning using `REPARTITION`/`REPARTITION_BY_RANGE` hint or `DISTRIBUTE BY`/`CLUSTER BY`, AQE will coalesce partitions. Unit tests. Closes apache#28952 from viirya/SPARK-32056-sql. Authored-by: Liang-Chi Hsieh <viirya@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
As the followup of #28900, this patch extends coalescing partitions to repartitioning using hints and SQL syntax without specifying number of partitions, when AQE is enabled.
Why are the changes needed?
When repartitionning using hints and SQL syntax, we should follow the shuffling behavior of repartition by expression/range to coalesce partitions when AQE is enabled.
Does this PR introduce any user-facing change?
Yes. After this change, if users don't specify the number of partitions when repartitioning using
REPARTITION/REPARTITION_BY_RANGEhint orDISTRIBUTE BY/CLUSTER BY, AQE will coalesce partitions.How was this patch tested?
Unit tests.