[BugFix] fix cte output column duplicate #34904

Seaven · 2023-11-14T02:19:08Z

In CTE's MultiCastDataSinks（exchange-3）, will prune column for CTE-consume, but set duplicate column, will crash when clone chunk

MySQL test_db> explain select * from t0
where (t0.v2 = 2) or t0.v3 in (select x0.v3 from t0 x0 where t0.v2 = x0.v3);
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Explain String                                                                                                                                                                                                                             |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                                                                                                                                                                            |
|  OUTPUT EXPRS:1: v1 | 2: v2 | 3: v3 | 4: v4 | 5: v5                                                                                                                                                                                        |
|   PARTITION: UNPARTITIONED                                                                                                                                                                                                                 |
|                                                                                                                                                                                                                                            |
|   RESULT SINK                                                                                                                                                                                                                              |
|                                                                                                                                                                                                                                            |
|   17:EXCHANGE                                                                                                                                                                                                                              |
|                                                                                                                                                                                                                                            |
| PLAN FRAGMENT 1                                                                                                                                                                                                                            |
|  OUTPUT EXPRS:                                                                                                                                                                                                                             |
|   PARTITION: HASH_PARTITIONED: 14: v3                                                                                                                                                                                                      |
|                                                                                                                                                                                                                                            |
|   STREAM DATA SINK                                                                                                                                                                                                                         |
|     EXCHANGE ID: 17                                                                                                                                                                                                                        |
|     UNPARTITIONED                                                                                                                                                                                                                          |
|                                                                                                                                                                                                                                            |
|   16:Project                                                                                                                                                                                                                               |
|   |  <slot 1> : 1: v1                                                                                                                                                                                                                      |
|   |  <slot 2> : 2: v2                                                                                                                                                                                                                      |
|   |  <slot 3> : 3: v3                                                                                                                                                                                                                      |
|   |  <slot 4> : 4: v4                                                                                                                                                                                                                      |
|   |  <slot 5> : 5: v5                                                                                                                                                                                                                      |
|   |                                                                                                                                                                                                                                        |
|   15:HASH JOIN                                                                                                                                                                                                                             |
|   |  join op: LEFT OUTER JOIN (BUCKET_SHUFFLE(S))                                                                                                                                                                                          |
|   |  colocate: false, reason:                                                                                                                                                                                                              |
|   |  equal join conjunct: 3: v3 = 12: v3                                                                                                                                                                                                   |
|   |  equal join conjunct: 2: v2 = 12: v3                                                                                                                                                                                                   |
|   |  other predicates: (2: v2 = 2) OR (CASE WHEN (15: countRows IS NULL) OR (15: countRows = 0) THEN FALSE WHEN 3: v3 IS NULL THEN NULL WHEN 12: v3 IS NOT NULL THEN TRUE WHEN 16: countNotNulls < 15: countRows THEN NULL ELSE FALSE END) |
|   |                                                                                                                                                                                                                                        |
|   |----14:EXCHANGE                                                                                                                                                                                                                         |
|   |                                                                                                                                                                                                                                        |
|   8:Project                                                                                                                                                                                                                                |
|   |  <slot 1> : 1: v1                                                                                                                                                                                                                      |
|   |  <slot 2> : 2: v2                                                                                                                                                                                                                      |
|   |  <slot 3> : 3: v3                                                                                                                                                                                                                      |
|   |  <slot 4> : 4: v4                                                                                                                                                                                                                      |
|   |  <slot 5> : 5: v5                                                                                                                                                                                                                      |
|   |  <slot 15> : 15: countRows                                                                                                                                                                                                             |
|   |  <slot 16> : 16: countNotNulls                                                                                                                                                                                                         |
|   |                                                                                                                                                                                                                                        |
|   7:HASH JOIN                                                                                                                                                                                                                              |
|   |  join op: LEFT OUTER JOIN (BUCKET_SHUFFLE(S))                                                                                                                                                                                          |
|   |  colocate: false, reason:                                                                                                                                                                                                              |
|   |  equal join conjunct: 2: v2 = 14: v3                                                                                                                                                                                                   |
|   |                                                                                                                                                                                                                                        |
|   |----6:AGGREGATE (update finalize)                                                                                                                                                                                                       |
|   |    |  output: count(1), count(13: v3)                                                                                                                                                                                                  |
|   |    |  group by: 14: v3                                                                                                                                                                                                                 |
|   |    |                                                                                                                                                                                                                                   |
|   |    5:EXCHANGE                                                                                                                                                                                                                          |
|   |                                                                                                                                                                                                                                        |
|   2:EXCHANGE                                                                                                                                                                                                                               |
|                                                                                                                                                                                                                                            |
| PLAN FRAGMENT 2                                                                                                                                                                                                                            |
|  OUTPUT EXPRS:                                                                                                                                                                                                                             |
|   PARTITION: HASH_PARTITIONED: 12: v3                                                                                                                                                                                                      |
|                                                                                                                                                                                                                                            |
|   STREAM DATA SINK                                                                                                                                                                                                                         |
|     EXCHANGE ID: 14                                                                                                                                                                                                                        |
|     HASH_PARTITIONED: 12: v3                                                                                                                                                                                                               |
|                                                                                                                                                                                                                                            |
|   13:AGGREGATE (merge finalize)                                                                                                                                                                                                            |
|   |  group by: 12: v3                                                                                                                                                                                                                      |
|   |                                                                                                                                                                                                                                        |
|   12:EXCHANGE                                                                                                                                                                                                                              |
|                                                                                                                                                                                                                                            |
| PLAN FRAGMENT 3                                                                                                                                                                                                                            |
|  OUTPUT EXPRS:                                                                                                                                                                                                                             |
|   PARTITION: RANDOM                                                                                                                                                                                                                        |
|                                                                                                                                                                                                                                            |
|   STREAM DATA SINK                                                                                                                                                                                                                         |
|     EXCHANGE ID: 12                                                                                                                                                                                                                        |
|     HASH_PARTITIONED: 12: v3                                                                                                                                                                                                               |
|                                                                                                                                                                                                                                            |
|   11:AGGREGATE (update serialize)                                                                                                                                                                                                          |
|   |  STREAMING                                                                                                                                                                                                                             |
|   |  group by: 12: v3                                                                                                                                                                                                                      |
|   |                                                                                                                                                                                                                                        |
|   10:Project                                                                                                                                                                                                                               |
|   |  <slot 12> : 8: v3                                                                                                                                                                                                                     |
|   |                                                                                                                                                                                                                                        |
|   9:EXCHANGE                                                                                                                                                                                                                               |
|                                                                                                                                                                                                                                            |
| PLAN FRAGMENT 4                                                                                                                                                                                                                            |
|  OUTPUT EXPRS:                                                                                                                                                                                                                             |
|   PARTITION: RANDOM                                                                                                                                                                                                                        |
|                                                                                                                                                                                                                                            |
|   STREAM DATA SINK                                                                                                                                                                                                                         |
|     EXCHANGE ID: 05                                                                                                                                                                                                                        |
|     HASH_PARTITIONED: 14: v3                                                                                                                                                                                                               |
|                                                                                                                                                                                                                                            |
|   4:Project                                                                                                                                                                                                                                |
|   |  <slot 13> : 8: v3                                                                                                                                                                                                                     |
|   |  <slot 14> : 8: v3                                                                                                                                                                                                                     |
|   |                                                                                                                                                                                                                                        |
|   3:EXCHANGE                                                                                                                                                                                                                               |
|                                                                                                                                                                                                                                            |
| PLAN FRAGMENT 5                                                                                                                                                                                                                            |
|  OUTPUT EXPRS:                                                                                                                                                                                                                             |
|   PARTITION: RANDOM                                                                                                                                                                                                                        |
|                                                                                                                                                                                                                                            |
|   STREAM DATA SINK                                                                                                                                                                                                                         |
|     EXCHANGE ID: 02                                                                                                                                                                                                                        |
|     HASH_PARTITIONED: 2: v2                                                                                                                                                                                                                |
|                                                                                                                                                                                                                                            |
|   1:OlapScanNode                                                                                                                                                                                                                           |
|      TABLE: t0                                                                                                                                                                                                                             |
|      PREAGGREGATION: ON                                                                                                                                                                                                                    |
|      partitions=1/1                                                                                                                                                                                                                        |
|      rollup: t0                                                                                                                                                                                                                            |
|      tabletRatio=3/3                                                                                                                                                                                                                       |
|      tabletList=7426956,7426958,7426960                                                                                                                                                                                                    |
|      cardinality=2                                                                                                                                                                                                                         |
|      avgRowSize=41.0                                                                                                                                                                                                                       |
|                                                                                                                                                                                                                                            |
| PLAN FRAGMENT 6                                                                                                                                                                                                                            |
|  OUTPUT EXPRS:8: v3                                                                                                                                                                                                                        |
|   PARTITION: RANDOM                                                                                                                                                                                                                        |
|                                                                                                                                                                                                                                            |
|   MultiCastDataSinks                                                                                                                                                                                                                       |
|   STREAM DATA SINK                                                                                                                                                                                                                         |
|     EXCHANGE ID: 03                                                                                                                                                                                                                        |
|     RANDOM                                                                                                                                                                                                                                 |
|   STREAM DATA SINK                                                                                                                                                                                                                         |
|     EXCHANGE ID: 09                                                                                                                                                                                                                        |
|     RANDOM                                                                                                                                                                                                                                 |
|                                                                                                                                                                                                                                            |
|   0:OlapScanNode                                                                                                                                                                                                                           |
|      TABLE: t0                                                                                                                                                                                                                             |
|      PREAGGREGATION: ON                                                                                                                                                                                                                    |
|      partitions=1/1                                                                                                                                                                                                                        |
|      rollup: t0                                                                                                                                                                                                                            |
|      tabletRatio=3/3                                                                                                                                                                                                                       |
|      tabletList=7426956,7426958,7426960                                                                                                                                                                                                    |
|      cardinality=2                                                                                                                                                                                                                         |
|      avgRowSize=8.0                                                                                                                                                                                                                        |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

Signed-off-by: Seaven <seaven_7@qq.com>

sonarqubecloud · 2023-11-15T03:57:15Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

0.0% Coverage
0.0% Duplication

The version of Java (11.0.21) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

github-actions · 2023-11-15T05:06:42Z

[FE Incremental Coverage Report]

✅ pass : 3 / 3 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/sql/plan/PlanFragmentBuilder.java	3	3	100.00%	[]

github-actions · 2023-11-15T05:07:58Z

[BE Incremental Coverage Report]

❌ fail : 0 / 5 (00.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	src/exec/pipeline/exchange/exchange_sink_operator.cpp	0	5	00.00%	[498, 499, 502, 503, 504]

github-actions · 2023-11-15T07:10:33Z

@Mergifyio backport branch-3.2

github-actions · 2023-11-15T07:10:35Z

@Mergifyio backport branch-3.1

mergify · 2023-11-15T07:10:37Z

backport branch-3.2

✅ Backports have been created

#35065 [BugFix] fix cte output column duplicate (backport #34904) has been created for branch branch-3.2

github-actions · 2023-11-15T07:10:37Z

@Mergifyio backport branch-3.0

github-actions · 2023-11-15T07:10:39Z

@Mergifyio backport branch-2.5

mergify · 2023-11-15T07:10:40Z

backport branch-3.1

✅ Backports have been created

#35066 [BugFix] fix cte output column duplicate (backport #34904) has been created for branch branch-3.1 but encountered conflicts

mergify · 2023-11-15T07:10:42Z

backport branch-3.0

✅ Backports have been created

#35067 [BugFix] fix cte output column duplicate (backport #34904) has been created for branch branch-3.0 but encountered conflicts

mergify · 2023-11-15T07:10:46Z

backport branch-2.5

✅ Backports have been created

#35068 [BugFix] fix cte output column duplicate (backport #34904) has been created for branch branch-2.5 but encountered conflicts

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit a2015c1)

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit a2015c1) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/sql/plan/PlanFragmentBuilder.java

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit a2015c1)

Signed-off-by: Seaven <seaven_7@qq.com>

… (StarRocks#35085) Signed-off-by: Seaven <seaven_7@qq.com>

mergify bot assigned Seaven Nov 14, 2023

github-actions bot added 3.2 3.1 3.0 2.5 labels Nov 14, 2023

[BugFix] fix cte output column lose schema

ffb181c

Signed-off-by: Seaven <seaven_7@qq.com>

Seaven force-pushed the fix3 branch from da03ee1 to ffb181c Compare November 15, 2023 03:50

Seaven changed the title ~~[BugFix] fix cte output column lose schema~~ [BugFix] fix cte output column duplicate Nov 15, 2023

kangkaisen approved these changes Nov 15, 2023

View reviewed changes

packy92 approved these changes Nov 15, 2023

View reviewed changes

packy92 merged commit a2015c1 into StarRocks:main Nov 15, 2023

github-actions bot removed the 3.2 label Nov 15, 2023

github-actions bot removed the 3.1 label Nov 15, 2023

github-actions bot removed the 3.0 label Nov 15, 2023

github-actions bot removed the 2.5 label Nov 15, 2023

mergify bot pushed a commit that referenced this pull request Nov 15, 2023

[BugFix] fix cte output column duplicate (#34904)

0c9ade3

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit a2015c1)

This was referenced Nov 15, 2023

[BugFix] fix cte output column duplicate (backport #34904) #35065

Merged

[BugFix] fix cte output column duplicate (backport #34904) #35066

Closed

mergify bot mentioned this pull request Nov 15, 2023

[BugFix] fix cte output column duplicate (backport #34904) #35067

Closed

mergify bot mentioned this pull request Nov 15, 2023

[BugFix] fix cte output column duplicate (backport #34904) #35068

Closed

wanpengfei-git pushed a commit that referenced this pull request Nov 15, 2023

[BugFix] fix cte output column duplicate (#34904)

5047357

Signed-off-by: Seaven <seaven_7@qq.com> (cherry picked from commit a2015c1)

Seaven added a commit to Seaven/starrocks that referenced this pull request Nov 15, 2023

[BugFix] fix cte output column duplicate (StarRocks#34904)

fffe463

Signed-off-by: Seaven <seaven_7@qq.com>

Seaven added a commit to Seaven/starrocks that referenced this pull request Nov 15, 2023

[BugFix] fix cte output column duplicate (StarRocks#34904)

60029a2

Signed-off-by: Seaven <seaven_7@qq.com>

wanpengfei-git pushed a commit that referenced this pull request Nov 15, 2023

[BugFix] fix cte output column duplicate (#34904)

a5fb2a9

Signed-off-by: Seaven <seaven_7@qq.com>

kangkaisen pushed a commit that referenced this pull request Nov 15, 2023

[BugFix][Cherry-Pick] fix cte output column duplicate (#34904) (#35086)

78c0107

Signed-off-by: Seaven <seaven_7@qq.com>

kangkaisen pushed a commit that referenced this pull request Nov 15, 2023

[BugFix][Cherry-Pick] fix cte output column duplicate (#34904) (#35085)

c440e39

Signed-off-by: Seaven <seaven_7@qq.com>

luohaha pushed a commit to luohaha/starrocks that referenced this pull request Nov 17, 2023

[BugFix][Cherry-Pick] fix cte output column duplicate (StarRocks#34904)…

2c1e6f2

… (StarRocks#35085) Signed-off-by: Seaven <seaven_7@qq.com>

luohaha mentioned this pull request Nov 17, 2023

[BugFix] Fix mishandled type null (backport #34985) #35334

Merged

22 tasks

Seaven deleted the fix3 branch December 5, 2023 12:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BugFix] fix cte output column duplicate #34904

[BugFix] fix cte output column duplicate #34904

Seaven commented Nov 14, 2023 •

edited

Loading

sonarqubecloud bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

mergify bot commented Nov 15, 2023 •

edited

Loading

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

mergify bot commented Nov 15, 2023 •

edited

Loading

mergify bot commented Nov 15, 2023 •

edited

Loading

mergify bot commented Nov 15, 2023 •

edited

Loading

[BugFix] fix cte output column duplicate #34904

[BugFix] fix cte output column duplicate #34904

Conversation

Seaven commented Nov 14, 2023 • edited Loading

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

sonarqubecloud bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

[FE Incremental Coverage Report]

file detail

github-actions bot commented Nov 15, 2023

[BE Incremental Coverage Report]

file detail

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

mergify bot commented Nov 15, 2023 • edited Loading

✅ Backports have been created

github-actions bot commented Nov 15, 2023

github-actions bot commented Nov 15, 2023

mergify bot commented Nov 15, 2023 • edited Loading

✅ Backports have been created

mergify bot commented Nov 15, 2023 • edited Loading

✅ Backports have been created

mergify bot commented Nov 15, 2023 • edited Loading

✅ Backports have been created

Seaven commented Nov 14, 2023 •

edited

Loading

mergify bot commented Nov 15, 2023 •

edited

Loading

mergify bot commented Nov 15, 2023 •

edited

Loading

mergify bot commented Nov 15, 2023 •

edited

Loading

mergify bot commented Nov 15, 2023 •

edited

Loading