Implement metrics for shuffle read and write #676
Conversation
| ahash = "0.7" | ||
| async-trait = "0.1.36" | ||
| futures = "0.3" | ||
| hashbrown = "0.11" |
There was a problem hiding this comment.
From the official readme:
Since Rust 1.36, this is now the HashMap implementation for the Rust standard library. However you may still want to use this crate instead since it works in environments without std, such as embedded systems and kernels.
Are we introducing it because we are expecting to use ballista in embedded systems?
There was a problem hiding this comment.
It looks like the dependency was added to DataFusion for a performance optimization in ec06207 and when the metrics method was added in DataFusion is used this hashmap as well, but perhaps we should have had it use the std hashmap instead?
@Dandandan what do you think?
There was a problem hiding this comment.
Filed #677 as a follow up so we don't block the merge of this PR. I am curious whether the std hashmap also uses ahash as the default hasher.
There was a problem hiding this comment.
Added some comments on the ticket.
3 participants
Which issue does this PR close?
Closes #675 .
Rationale for this change
What changes are included in this PR?
More metrics.
Are there any user-facing changes?
More metrics in query plan.