Skip to content

Conversation

@alamb
Copy link
Contributor

@alamb alamb commented Aug 31, 2021

Which issue does this PR close?

Next part #866

Rationale for this change

We want basic understanding of where a plan's time is spent and in what operators. See #866 for more details

What changes are included in this PR?

  1. Instrument FilterExec using the API from Add BaselineMetrics, Timestamp metrics, add for CoalescePartitionsExec, rename output_time -> elapsed_compute #909
  2. Tests for same

Are there any user-facing changes?

More fields in EXPLAIN ANALYZE are now filled out

Example of how explain analayze is looking:

EXPLAIN ANALYZE select count(*) from (SELECT count(*), c1 FROM aggregate_test_100 WHERE c13 != 'C2GT5KVyOPZpgKVl110TyZO0NcJ434' GROUP BY c1 ORDER BY c1)


+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type         | plan                                                                                                                                                                                                                              |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | ProjectionExec: expr=[COUNT(UInt8(1))@0 as COUNT(UInt8(1))], metrics=[]                                                                                                                                                           |
|                   |   HashAggregateExec: mode=Final, gby=[], aggr=[COUNT(UInt8(1))], metrics=[output_rows=1, elapsed_compute=69.362µs]                                                                                                                |
|                   |     CoalescePartitionsExec, metrics=[output_rows=3, elapsed_compute=NOT RECORDED]                                                                                                                                                 |
|                   |       HashAggregateExec: mode=Partial, gby=[], aggr=[COUNT(UInt8(1))], metrics=[output_rows=3, elapsed_compute=119.041µs]                                                                                                         |
|                   |         RepartitionExec: partitioning=RoundRobinBatch(3), metrics=[send_time{inputPartition=0}=4.775µs, fetch_time{inputPartition=0}=12.77346ms, repart_time{inputPartition=0}=NOT RECORDED]                                      |
|                   |           SortExec: [c1@0 ASC], metrics=[output_rows=5, elapsed_compute=232.203µs]                                                                                                                                                |
|                   |             CoalescePartitionsExec, metrics=[output_rows=5, elapsed_compute=NOT RECORDED]                                                                                                                                         |
|                   |               HashAggregateExec: mode=FinalPartitioned, gby=[c1@0 as c1], aggr=[COUNT(UInt8(1))], metrics=[output_rows=5, elapsed_compute=373.876µs]                                                                              |
|                   |                 CoalesceBatchesExec: target_batch_size=4096, metrics=[]                                                                                                                                                           |
|                   |                   RepartitionExec: partitioning=Hash([Column { name: "c1", index: 0 }], 3), metrics=[fetch_time{inputPartition=0}=34.072102ms, repart_time{inputPartition=0}=252.872µs, send_time{inputPartition=0}=NOT RECORDED] |
|                   |                     HashAggregateExec: mode=Partial, gby=[c1@0 as c1], aggr=[COUNT(UInt8(1))], metrics=[output_rows=5, elapsed_compute=581.405µs]                                                                                 |
|                   |                       CoalesceBatchesExec: target_batch_size=4096, metrics=[]                                                                                                                                                     |
|                   |                         FilterExec: c13@1 != C2GT5KVyOPZpgKVl110TyZO0NcJ434, metrics=[output_rows=99, elapsed_compute=309.095µs]                                                                                                  |
|                   |                           RepartitionExec: partitioning=RoundRobinBatch(3), metrics=[fetch_time{inputPartition=0}=8.239528ms, send_time{inputPartition=0}=9.864µs, repart_time{inputPartition=0}=NOT RECORDED]                    |
|                   |                             CsvExec: source=Path(ARROW_TEST_DATA/csv/aggregate_test_100.csv: [ARROW_TEST_DATA/csv/aggregate_test_100.csv]), has_header=true, metrics=[]         |
+-------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

@alamb alamb force-pushed the alamb/filter_stats branch from 8d4077d to 299f795 Compare September 9, 2021 18:11
@alamb
Copy link
Contributor Author

alamb commented Sep 9, 2021

clippy failure is due to #986 I think

@alamb alamb force-pushed the alamb/filter_stats branch from 299f795 to 45c1b0f Compare September 9, 2021 20:59
@alamb alamb merged commit 4c0d430 into apache:master Sep 10, 2021
@alamb alamb deleted the alamb/filter_stats branch September 10, 2021 17:07
@houqp houqp added the enhancement New feature or request label Nov 6, 2021
unkloud pushed a commit to unkloud/datafusion that referenced this pull request Mar 23, 2025
## Which issue does this PR close?
Closes apache#521 and apache#665.

## What changes are included in this PR?
This PR normalize the right hand of div and rem

## How are these changes tested?
Test updated
H0TB0X420 pushed a commit to H0TB0X420/datafusion that referenced this pull request Oct 7, 2025
* Generate changelog

* cargo update
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants