[EPIC] Improve `metrics` tracking in `EXPLAIN ANALYZE`

[2010YOUY01]

Is your feature request related to a problem or challenge?

This issue is to track ideas for improving metrics tracking, such as implementing useful metrics that are currently missing.

• Add output_bytes metrics to Explain Analyze #16244
• Inconsistent time_elapsed_scanning_total result across different queries scanning the same data #16619
• Add a fine grain memory usage tracking / break down of memory usage *within* each operator #16904
• MetricValue::Custom encoding\decoding support #16945
• Include metric output_batches into BaselineMetrics #17027
• Epic: Decides default metrics to show in EXPLAIN ANALYZE across operators #18116
• Improve metrics in DataSourceExec with Parquet source #18195
• RepartitionExec is missing BaselineMetrics #18218
• feat: Add selectivity metrics to FilterExec #18406
• Support reduction_fator metric (for EXPLAIN ANALYZE) in AggregateExec #18410
• Add selectivity metrics (for Explain Analyze) in Hash Join #18409
• Support selectivity metrics (for Explain Analyze) in Nested Loop Join #18407
• Track individual expr's execution time in ProjectExec metrics (in EXPLAIN ANALYZE) #18456
• Minor enhancements to EXPLAIN ANALYZE output formatting #18689
• Elapsed compute time not tracked for csv file scan #18795
• feat(parquet): Implement scan_efficiency_ratio metric for parquet reading #18577

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

[simonvandel]

ProjectionExec currently shows the total elapsed time for all expressions. Would it make sense to show timings for each individual expression? Perhaps we could do it in VERBOSE mode

[alamb]

Should we mention the improvements to explain analyze in the blog post for 51?

There are some neat improvements I think

I have a draf here

• Blog post for DataFusion 51.0.0 datafusion-site#124

[2010YOUY01]

Should we mention the improvements to explain analyze in the blog post for 51?

There are some neat improvements I think

I have a draf here

• Add blog post for DataFusion 51.0.0 datafusion-site#124

I have added more details to the draft blogpost.

There are several other useful metric additions that will be included in the upcoming 52.0.0 release. Also, the user guide (https://datafusion.apache.org/user-guide/metrics.html) hasn’t been updated yet, so let’s continue improving it and ship more updates in the next release 😄

[alamb]

Sounds like a great plan!