Skip to content

Commit 9a5eaa5

Browse files
cloud-fandongjoon-hyun
cloud-fan
authored and
dongjoon-hyun
committed
[SPARK-39437][SQL][TEST][3.3] Normalize plan id separately in PlanStabilitySuite
### What changes were proposed in this pull request? In `PlanStabilitySuite`, we normalize expression IDs by matching `#\d+` in the explain string. However, this regex can match plan id in `Exchange` node as well, which will mess up the normalization if expression IDs and plan IDs overlap. This PR normalizes plan id separately in `PlanStabilitySuite`. ### Why are the changes needed? Make the plan golden file more stable. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? N/A backport #36827 Closes #36854 from cloud-fan/test2. Authored-by: Wenchen Fan <wenchen@databricks.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent bcc646b commit 9a5eaa5

File tree

331 files changed

+27432
-27422
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

331 files changed

+27432
-27422
lines changed

‎sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/Exchange.scala

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ abstract class Exchange extends UnaryExecNode {
3737
override def output: Seq[Attribute] = child.output
3838
final override val nodePatterns: Seq[TreePattern] = Seq(EXCHANGE)
3939

40-
override def stringArgs: Iterator[Any] = super.stringArgs ++ Iterator(s"[id=#$id]")
40+
override def stringArgs: Iterator[Any] = super.stringArgs ++ Iterator(s"[plan_id=$id]")
4141
}
4242

4343
/**

‎sql/core/src/test/resources/sql-tests/results/explain-aqe.sql.out

Lines changed: 21 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -67,10 +67,10 @@ Aggregate [sum(distinct val#x) AS sum(DISTINCT val)#xL]
6767
== Physical Plan ==
6868
AdaptiveSparkPlan isFinalPlan=false
6969
+- HashAggregate(keys=[], functions=[sum(distinct val#x)], output=[sum(DISTINCT val)#xL])
70-
+- Exchange SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
70+
+- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
7171
+- HashAggregate(keys=[], functions=[partial_sum(distinct val#x)], output=[sum#xL])
7272
+- HashAggregate(keys=[val#x], functions=[], output=[val#x])
73-
+- Exchange hashpartitioning(val#x, 4), ENSURE_REQUIREMENTS, [id=#x]
73+
+- Exchange hashpartitioning(val#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
7474
+- HashAggregate(keys=[val#x], functions=[], output=[val#x])
7575
+- FileScan parquet default.explain_temp1[val#x] Batched: true, DataFilters: [], Format: Parquet, Location [not included in comparison]/{warehouse_dir}/explain_temp1], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<val:int>
7676

@@ -116,7 +116,7 @@ Results [2]: [key#x, max#x]
116116

117117
(4) Exchange
118118
Input [2]: [key#x, max#x]
119-
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [id=#x]
119+
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
120120

121121
(5) HashAggregate
122122
Input [2]: [key#x, max#x]
@@ -127,7 +127,7 @@ Results [2]: [key#x, max(val#x)#x AS max(val)#x]
127127

128128
(6) Exchange
129129
Input [2]: [key#x, max(val)#x]
130-
Arguments: rangepartitioning(key#x ASC NULLS FIRST, 4), ENSURE_REQUIREMENTS, [id=#x]
130+
Arguments: rangepartitioning(key#x ASC NULLS FIRST, 4), ENSURE_REQUIREMENTS, [plan_id=x]
131131

132132
(7) Sort
133133
Input [2]: [key#x, max(val)#x]
@@ -178,7 +178,7 @@ Results [2]: [key#x, max#x]
178178

179179
(4) Exchange
180180
Input [2]: [key#x, max#x]
181-
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [id=#x]
181+
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
182182

183183
(5) HashAggregate
184184
Input [2]: [key#x, max#x]
@@ -249,7 +249,7 @@ Results [2]: [key#x, val#x]
249249

250250
(7) Exchange
251251
Input [2]: [key#x, val#x]
252-
Arguments: hashpartitioning(key#x, val#x, 4), ENSURE_REQUIREMENTS, [id=#x]
252+
Arguments: hashpartitioning(key#x, val#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
253253

254254
(8) HashAggregate
255255
Input [2]: [key#x, val#x]
@@ -306,7 +306,7 @@ Condition : isnotnull(key#x)
306306

307307
(5) BroadcastExchange
308308
Input [2]: [key#x, val#x]
309-
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [id=#x]
309+
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=x]
310310

311311
(6) BroadcastHashJoin
312312
Left keys [1]: [key#x]
@@ -355,7 +355,7 @@ Condition : isnotnull(key#x)
355355

356356
(4) BroadcastExchange
357357
Input [2]: [key#x, val#x]
358-
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [id=#x]
358+
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=x]
359359

360360
(5) BroadcastHashJoin
361361
Left keys [1]: [key#x]
@@ -438,7 +438,7 @@ Results [1]: [max#x]
438438

439439
(8) Exchange
440440
Input [1]: [max#x]
441-
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
441+
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
442442

443443
(9) HashAggregate
444444
Input [1]: [max#x]
@@ -485,7 +485,7 @@ Results [1]: [max#x]
485485

486486
(15) Exchange
487487
Input [1]: [max#x]
488-
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
488+
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
489489

490490
(16) HashAggregate
491491
Input [1]: [max#x]
@@ -569,7 +569,7 @@ Results [1]: [max#x]
569569

570570
(8) Exchange
571571
Input [1]: [max#x]
572-
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
572+
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
573573

574574
(9) HashAggregate
575575
Input [1]: [max#x]
@@ -616,7 +616,7 @@ Results [2]: [sum#x, count#xL]
616616

617617
(15) Exchange
618618
Input [2]: [sum#x, count#xL]
619-
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
619+
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
620620

621621
(16) HashAggregate
622622
Input [2]: [sum#x, count#xL]
@@ -682,7 +682,7 @@ Results [2]: [sum#x, count#xL]
682682

683683
(6) Exchange
684684
Input [2]: [sum#x, count#xL]
685-
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
685+
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
686686

687687
(7) HashAggregate
688688
Input [2]: [sum#x, count#xL]
@@ -718,7 +718,7 @@ Results [2]: [sum#x, count#xL]
718718

719719
(11) Exchange
720720
Input [2]: [sum#x, count#xL]
721-
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
721+
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
722722

723723
(12) HashAggregate
724724
Input [2]: [sum#x, count#xL]
@@ -777,7 +777,7 @@ Condition : (isnotnull(key#x) AND (key#x > 10))
777777

778778
(5) BroadcastExchange
779779
Input [2]: [key#x, val#x]
780-
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [id=#x]
780+
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=x]
781781

782782
(6) BroadcastHashJoin
783783
Left keys [1]: [key#x]
@@ -837,7 +837,7 @@ Results [2]: [key#x, max#x]
837837

838838
(4) Exchange
839839
Input [2]: [key#x, max#x]
840-
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [id=#x]
840+
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
841841

842842
(5) HashAggregate
843843
Input [2]: [key#x, max#x]
@@ -866,7 +866,7 @@ Results [2]: [key#x, max#x]
866866

867867
(9) Exchange
868868
Input [2]: [key#x, max#x]
869-
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [id=#x]
869+
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
870870

871871
(10) HashAggregate
872872
Input [2]: [key#x, max#x]
@@ -877,7 +877,7 @@ Results [2]: [key#x, max(val#x)#x AS max(val)#x]
877877

878878
(11) BroadcastExchange
879879
Input [2]: [key#x, max(val)#x]
880-
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [id=#x]
880+
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [plan_id=x]
881881

882882
(12) BroadcastHashJoin
883883
Left keys [1]: [key#x]
@@ -964,7 +964,7 @@ Results [3]: [count#xL, sum#xL, count#xL]
964964

965965
(3) Exchange
966966
Input [3]: [count#xL, sum#xL, count#xL]
967-
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
967+
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
968968

969969
(4) HashAggregate
970970
Input [3]: [count#xL, sum#xL, count#xL]
@@ -1009,7 +1009,7 @@ Results [2]: [key#x, buf#x]
10091009

10101010
(3) Exchange
10111011
Input [2]: [key#x, buf#x]
1012-
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [id=#x]
1012+
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
10131013

10141014
(4) ObjectHashAggregate
10151015
Input [2]: [key#x, buf#x]
@@ -1060,7 +1060,7 @@ Results [2]: [key#x, min#x]
10601060

10611061
(4) Exchange
10621062
Input [2]: [key#x, min#x]
1063-
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [id=#x]
1063+
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
10641064

10651065
(5) Sort
10661066
Input [2]: [key#x, min#x]

‎sql/core/src/test/resources/sql-tests/results/explain-cbo.sql.out

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ AdaptiveSparkPlan isFinalPlan=false
7272
: +- HashAggregate(keys=[], functions=[max(csales#xL)], output=[tpcds_cmax#xL])
7373
: +- HashAggregate(keys=[], functions=[partial_max(csales#xL)], output=[max#xL])
7474
: +- HashAggregate(keys=[], functions=[sum(b#x)], output=[csales#xL])
75-
: +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
75+
: +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
7676
: +- HashAggregate(keys=[], functions=[partial_sum(b#x)], output=[sum#xL])
7777
: +- Project [b#x]
7878
: +- Filter (isnotnull(a#x) AND (a#x < 100))

‎sql/core/src/test/resources/sql-tests/results/explain.sql.out

Lines changed: 19 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -66,10 +66,10 @@ Aggregate [sum(distinct val#x) AS sum(DISTINCT val)#xL]
6666

6767
== Physical Plan ==
6868
*HashAggregate(keys=[], functions=[sum(distinct val#x)], output=[sum(DISTINCT val)#xL])
69-
+- Exchange SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
69+
+- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
7070
+- *HashAggregate(keys=[], functions=[partial_sum(distinct val#x)], output=[sum#xL])
7171
+- *HashAggregate(keys=[val#x], functions=[], output=[val#x])
72-
+- Exchange hashpartitioning(val#x, 4), ENSURE_REQUIREMENTS, [id=#x]
72+
+- Exchange hashpartitioning(val#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
7373
+- *HashAggregate(keys=[val#x], functions=[], output=[val#x])
7474
+- *ColumnarToRow
7575
+- FileScan parquet default.explain_temp1[val#x] Batched: true, DataFilters: [], Format: Parquet, Location [not included in comparison]/{warehouse_dir}/explain_temp1], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<val:int>
@@ -119,7 +119,7 @@ Results [2]: [key#x, max#x]
119119

120120
(5) Exchange
121121
Input [2]: [key#x, max#x]
122-
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [id=#x]
122+
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
123123

124124
(6) HashAggregate [codegen id : 2]
125125
Input [2]: [key#x, max#x]
@@ -130,7 +130,7 @@ Results [2]: [key#x, max(val#x)#x AS max(val)#x]
130130

131131
(7) Exchange
132132
Input [2]: [key#x, max(val)#x]
133-
Arguments: rangepartitioning(key#x ASC NULLS FIRST, 4), ENSURE_REQUIREMENTS, [id=#x]
133+
Arguments: rangepartitioning(key#x ASC NULLS FIRST, 4), ENSURE_REQUIREMENTS, [plan_id=x]
134134

135135
(8) Sort [codegen id : 3]
136136
Input [2]: [key#x, max(val)#x]
@@ -180,7 +180,7 @@ Results [2]: [key#x, max#x]
180180

181181
(5) Exchange
182182
Input [2]: [key#x, max#x]
183-
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [id=#x]
183+
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
184184

185185
(6) HashAggregate [codegen id : 2]
186186
Input [2]: [key#x, max#x]
@@ -254,7 +254,7 @@ Results [2]: [key#x, val#x]
254254

255255
(9) Exchange
256256
Input [2]: [key#x, val#x]
257-
Arguments: hashpartitioning(key#x, val#x, 4), ENSURE_REQUIREMENTS, [id=#x]
257+
Arguments: hashpartitioning(key#x, val#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
258258

259259
(10) HashAggregate [codegen id : 4]
260260
Input [2]: [key#x, val#x]
@@ -314,7 +314,7 @@ Condition : isnotnull(key#x)
314314

315315
(7) BroadcastExchange
316316
Input [2]: [key#x, val#x]
317-
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [id=#x]
317+
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=x]
318318

319319
(8) BroadcastHashJoin [codegen id : 2]
320320
Left keys [1]: [key#x]
@@ -366,7 +366,7 @@ Condition : isnotnull(key#x)
366366

367367
(6) BroadcastExchange
368368
Input [2]: [key#x, val#x]
369-
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [id=#x]
369+
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=x]
370370

371371
(7) BroadcastHashJoin [codegen id : 2]
372372
Left keys [1]: [key#x]
@@ -447,7 +447,7 @@ Results [1]: [max#x]
447447

448448
(9) Exchange
449449
Input [1]: [max#x]
450-
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
450+
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
451451

452452
(10) HashAggregate [codegen id : 2]
453453
Input [1]: [max#x]
@@ -493,7 +493,7 @@ Results [1]: [max#x]
493493

494494
(16) Exchange
495495
Input [1]: [max#x]
496-
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
496+
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
497497

498498
(17) HashAggregate [codegen id : 2]
499499
Input [1]: [max#x]
@@ -575,7 +575,7 @@ Results [1]: [max#x]
575575

576576
(9) Exchange
577577
Input [1]: [max#x]
578-
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
578+
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
579579

580580
(10) HashAggregate [codegen id : 2]
581581
Input [1]: [max#x]
@@ -621,7 +621,7 @@ Results [2]: [sum#x, count#xL]
621621

622622
(16) Exchange
623623
Input [2]: [sum#x, count#xL]
624-
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
624+
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
625625

626626
(17) HashAggregate [codegen id : 2]
627627
Input [2]: [sum#x, count#xL]
@@ -685,7 +685,7 @@ Results [2]: [sum#x, count#xL]
685685

686686
(7) Exchange
687687
Input [2]: [sum#x, count#xL]
688-
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
688+
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
689689

690690
(8) HashAggregate [codegen id : 2]
691691
Input [2]: [sum#x, count#xL]
@@ -749,7 +749,7 @@ Condition : (isnotnull(key#x) AND (key#x > 10))
749749

750750
(7) BroadcastExchange
751751
Input [2]: [key#x, val#x]
752-
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [id=#x]
752+
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=x]
753753

754754
(8) BroadcastHashJoin [codegen id : 2]
755755
Left keys [1]: [key#x]
@@ -805,7 +805,7 @@ Results [2]: [key#x, max#x]
805805

806806
(5) Exchange
807807
Input [2]: [key#x, max#x]
808-
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [id=#x]
808+
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
809809

810810
(6) HashAggregate [codegen id : 4]
811811
Input [2]: [key#x, max#x]
@@ -826,7 +826,7 @@ Results [2]: [key#x, max(val#x)#x AS max(val)#x]
826826

827827
(9) BroadcastExchange
828828
Input [2]: [key#x, max(val)#x]
829-
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [id=#x]
829+
Arguments: HashedRelationBroadcastMode(List(cast(input[0, int, true] as bigint)),false), [plan_id=x]
830830

831831
(10) BroadcastHashJoin [codegen id : 4]
832832
Left keys [1]: [key#x]
@@ -912,7 +912,7 @@ Results [3]: [count#xL, sum#xL, count#xL]
912912

913913
(4) Exchange
914914
Input [3]: [count#xL, sum#xL, count#xL]
915-
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [id=#x]
915+
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=x]
916916

917917
(5) HashAggregate [codegen id : 2]
918918
Input [3]: [count#xL, sum#xL, count#xL]
@@ -956,7 +956,7 @@ Results [2]: [key#x, buf#x]
956956

957957
(4) Exchange
958958
Input [2]: [key#x, buf#x]
959-
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [id=#x]
959+
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
960960

961961
(5) ObjectHashAggregate
962962
Input [2]: [key#x, buf#x]
@@ -1006,7 +1006,7 @@ Results [2]: [key#x, min#x]
10061006

10071007
(5) Exchange
10081008
Input [2]: [key#x, min#x]
1009-
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [id=#x]
1009+
Arguments: hashpartitioning(key#x, 4), ENSURE_REQUIREMENTS, [plan_id=x]
10101010

10111011
(6) Sort [codegen id : 2]
10121012
Input [2]: [key#x, min#x]

0 commit comments

Comments
 (0)