Implement partition_statistics API for InterleaveExec
#17051
Conversation
github-actions
bot
added
core
liamzwbao
force-pushed
the
issue-15873-interleave
branch
from
9f4ddfe to
e74a202
Compare
liamzwbao
force-pushed
the
issue-15873-interleave
branch
from
e74a202 to
58f51c7
Compare
|
Hi @xudong963, this PR is ready for review |
|
@liamzwbao Sorry, I missed the PR, will review later. |
| assert_eq!(partition_row_counts[0], 2); | ||
| assert_eq!(partition_row_counts[1], 6); |
There was a problem hiding this comment.
I saw the real stat outputs are different from expected_stats, could you please help me figure out what results in the difference?
There was a problem hiding this comment.
The reason is that hash repartitioning doesn’t always distribute results evenly, which is expected and should have only a minor impact when the underlying dataset is large enough. I added a test for repartition to confirm this behavior. Note that the stats in repartition are marked as Inexact because the partitioning algorithm does not guarantee balanced output, and Interleave simply converges results from the child partitions.
Sorry for the late reply, I was out for the past 2 weeks
2 participants
Which issue does this PR close?
partition_statisticsAPI for more operators #15873.Rationale for this change
What changes are included in this PR?
Implement
partition_statisticsAPI forInterleaveExecAre these changes tested?
Yes
Are there any user-facing changes?
No