docs: add HashJoinExec metrics to user guide - #24055
Conversation
|
Hi maintainers, just a gentle follow-up on this documentation PR. I have rechecked the metric semantics against the implementation and local |
kosiew
left a comment
There was a problem hiding this comment.
Thanks for adding the HashJoinExec metrics documentation and keeping the BuildProbeJoinMetrics comments in sync. This looks good overall. I just have one small wording suggestion to make the memory metric a bit more precise.
| | build_time | Total time spent collecting and building the build side of the join. | | ||
| | build_input_batches | Number of input batches consumed from the build side. | | ||
| | build_input_rows | Number of input rows consumed from the build side. | | ||
| | build_mem_used | Peak memory used by the build side, in bytes. | |
There was a problem hiding this comment.
Could we say "Peak tracked memory used by the build side" instead of "Peak memory used" here? build_mem_used is based on accounting from RecordBatch sizes, ArrayMap and hash-table estimates, and the visited bitmap. That makes it useful for the operator's reservation and accounting, but it is not an allocator-measured process-memory peak.
Which issue does this PR close?
HashJoinExecin the metrics user-guide #19044.Rationale for this change
HashJoinExecexposes several operator-specific metrics, but the metrics user guide did not document them yet. This PR follows the metrics documentation tracking issue by adding theHashJoinExecsection.What changes are included in this PR?
HashJoinExecoperator-specific metrics todocs/source/user-guide/metrics.md.BuildProbeJoinMetricsto keep metric descriptions in sync with the user guide.join_timecovers join processing after build-side collection.probe_hit_rateandavg_fanoutmeasure join-key matches before applying any join filter.Are these changes tested?
This is a documentation/comment-only change with no runtime behavior change.
Checked locally:
git diff --checkcargo fmt --checknpx prettier@2.7.1 --check docs/source/user-guide/metrics.md./dev/rust_lint.shcd docs && uv run --package datafusion-docs ./build.shEXPLAIN ANALYZEqueries with and without a join filterAre there any user-facing changes?
Yes. The metrics user guide now documents the existing
HashJoinExecmetrics. There are no runtime behavior or public API changes.AI-assisted contribution note
This change is limited to documentation and comments. I verified the metric names and semantics against
BuildProbeJoinMetrics, theHashJoinExecexecution path, and localEXPLAIN ANALYZEoutput. I do not have unknowns or assumptions to call out.