-
Notifications
You must be signed in to change notification settings - Fork 1.5k
test: Port (last) repartition.rs
query to sqllogictest
#8936
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -71,3 +71,59 @@ AggregateExec: mode=FinalPartitioned, gby=[column1@0 as column1], aggr=[SUM(parq | |
# Cleanup | ||
statement ok | ||
DROP TABLE parquet_table; | ||
|
||
|
||
|
||
# Unbounded repartition | ||
# See https://github.com/apache/arrow-datafusion/issues/5278 | ||
# Set up unbounded table and run a query - the query plan should display a `RepartitionExec` | ||
# and a `CoalescePartitionsExec` | ||
statement ok | ||
CREATE UNBOUNDED EXTERNAL TABLE sink_table ( | ||
c1 VARCHAR NOT NULL, | ||
c2 TINYINT NOT NULL, | ||
c3 SMALLINT NOT NULL, | ||
c4 SMALLINT NOT NULL, | ||
c5 INTEGER NOT NULL, | ||
c6 BIGINT NOT NULL, | ||
c7 SMALLINT NOT NULL, | ||
c8 INT NOT NULL, | ||
c9 INT UNSIGNED NOT NULL, | ||
c10 BIGINT UNSIGNED NOT NULL, | ||
c11 FLOAT NOT NULL, | ||
c12 DOUBLE NOT NULL, | ||
c13 VARCHAR NOT NULL | ||
) | ||
STORED AS CSV | ||
WITH HEADER ROW | ||
LOCATION '../../testing/data/csv/aggregate_test_100.csv'; | ||
|
||
query TII | ||
SELECT c1, c2, c3 FROM sink_table WHERE c3 > 0 LIMIT 5; | ||
---- | ||
c 2 1 | ||
b 1 29 | ||
e 3 104 | ||
a 3 13 | ||
d 1 38 | ||
|
||
statement ok | ||
set datafusion.execution.target_partitions = 3; | ||
|
||
statement ok | ||
set datafusion.optimizer.enable_round_robin_repartition = true; | ||
|
||
query TT | ||
EXPLAIN SELECT c1, c2, c3 FROM sink_table WHERE c3 > 0 LIMIT 5; | ||
---- | ||
logical_plan | ||
Limit: skip=0, fetch=5 | ||
--Filter: sink_table.c3 > Int16(0) | ||
----TableScan: sink_table projection=[c1, c2, c3] | ||
physical_plan | ||
GlobalLimitExec: skip=0, fetch=5 | ||
--CoalescePartitionsExec | ||
----CoalesceBatchesExec: target_batch_size=8192 | ||
------FilterExec: c3@2 > 0 | ||
--------RepartitionExec: partitioning=RoundRobinBatch(3), input_partitions=1 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This partitioning is round robin partitioning where the original test uses hash partitioning. I am not sure if that is equivalent. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I see! Thanks for looking into it There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think there will be a difference. It's hard to reproduce the hash partitioning with unbounded tables. However, this would work well enough. |
||
----------StreamingTableExec: partition_sizes=1, projection=[c1, c2, c3], infinite_source=true | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 |
Uh oh!
There was an error while loading. Please reload this page.