Skip to content

Conversation

@jonahgao
Copy link
Member

@jonahgao jonahgao commented Apr 4, 2024

Which issue does this PR close?

Closes #9930.

Rationale for this change

The RepartitionExec above RecursiveQueryExec may cause the result of this query to be non-deterministic.

DataFusion CLI v37.0.0
❯ explain WITH RECURSIVE my_cte AS(
    SELECT 1::int AS a
    UNION ALL
    SELECT a::bigint+2 FROM my_cte WHERE a<3
) SELECT *, arrow_typeof(a) FROM my_cte;
+---------------+---------------------------------------------------------------------------------------------+
| plan_type     | plan                                                                                        |
+---------------+---------------------------------------------------------------------------------------------+
| physical_plan | ProjectionExec: expr=[a@0 as a, arrow_typeof(a@0) as arrow_typeof(my_cte.a)]                |
|               |   RepartitionExec: partitioning=RoundRobinBatch(12), input_partitions=1                     |
|               |     RecursiveQueryExec: name=my_cte, is_distinct=false                                      |
|               |       ProjectionExec: expr=[1 as a]                                                         |
|               |         PlaceholderRowExec                                                                  |
|               |       CoalescePartitionsExec                                                                |
|               |         ProjectionExec: expr=[CAST(CAST(a@0 AS Int64) + 2 AS Int32) as my_cte.a + Int64(2)] |
|               |           CoalesceBatchesExec: target_batch_size=8192                                       |
|               |             FilterExec: a@0 < 3                                                             |
|               |               RepartitionExec: partitioning=RoundRobinBatch(12), input_partitions=1         |
|               |                 WorkTableExec: name=my_cte                                                  |
|               |                                                                                             |
+---------------+---------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 0.018 seconds.

What changes are included in this PR?

Add rowsort to make the query result deterministic.

Are these changes tested?

Yes

Are there any user-facing changes?

No

@github-actions github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Apr 4, 2024
Copy link
Contributor

@comphead comphead left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm thanks @jonahgao nice catch

@alamb
Copy link
Contributor

alamb commented Apr 4, 2024

FYI @peter-toth

Thanks @jonahgao and @comphead

@alamb alamb merged commit c47a80a into apache:main Apr 4, 2024
@jonahgao jonahgao deleted the cte_test branch April 4, 2024 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Intermittent failure in test_files/cte.slt:751

3 participants