Commit c6be207
[SPARK-32616][SQL] Window operators should be added determinedly
### What changes were proposed in this pull request?
Use the `LinkedHashMap` instead of `immutable.Map` to hold the `Window` expressions in `ExtractWindowExpressions.addWindow`.
### Why are the changes needed?
This is a bug fix for apache#29270. In that PR, the generated plan(especially for the queries q47, q49, q57) on Jenkins always can not match the golden plan generated on my laptop.
It happens because `ExtractWindowExpressions.addWindow` now uses `immutable.Map` to hold the `Window` expressions by the key `(spec.partitionSpec, spec.orderSpec, WindowFunctionType.functionType(expr))` and converts the map to `Seq` at the end. Then, the `Seq` is used to add Window operators on top of the child plan. However, for the same query, the order of Windows expression inside the `Seq` could be undetermined when the expression id changes(which can affect the key). As a result, the same query could have different plans because of the undetermined order of Window operators.
Therefore, we use `LinkedHashMap`, which records the insertion order of entries, to make the adding order determined.
### Does this PR introduce _any_ user-facing change?
Maybe yes, users now always see the same plan for the same queries with multiple Window operators.
### How was this patch tested?
It's really hard to make a reproduce demo. I just tested manually with apache#29270 and it looks good.
Closes apache#29432 from Ngone51/fix-addWindow.
Authored-by: yi.wu <yi.wu@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>1 parent 10edeaf commit c6be207
File tree
1 file changed
+11
-3
lines changed- sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis
1 file changed
+11
-3
lines changedLines changed: 11 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2552 | 2552 | | |
2553 | 2553 | | |
2554 | 2554 | | |
| 2555 | + | |
| 2556 | + | |
2555 | 2557 | | |
2556 | 2558 | | |
2557 | 2559 | | |
| |||
2696 | 2698 | | |
2697 | 2699 | | |
2698 | 2700 | | |
| 2701 | + | |
| 2702 | + | |
| 2703 | + | |
2699 | 2704 | | |
2700 | | - | |
| 2705 | + | |
2701 | 2706 | | |
2702 | 2707 | | |
2703 | 2708 | | |
| |||
2713 | 2718 | | |
2714 | 2719 | | |
2715 | 2720 | | |
2716 | | - | |
| 2721 | + | |
| 2722 | + | |
| 2723 | + | |
| 2724 | + | |
2717 | 2725 | | |
2718 | | - | |
| 2726 | + | |
2719 | 2727 | | |
2720 | 2728 | | |
2721 | 2729 | | |
| |||
0 commit comments