Skip to content

Fix incorrect Celeborn mapStatus#1133

Merged
richox merged 1 commit intoapache:masterfrom
cxzl25:celeborn_mapstatus
Aug 12, 2025
Merged

Fix incorrect Celeborn mapStatus#1133
richox merged 1 commit intoapache:masterfrom
cxzl25:celeborn_mapstatus

Conversation

@cxzl25
Copy link
Contributor

@cxzl25 cxzl25 commented Aug 8, 2025

Which issue does this PR close?

Closes #.

Rationale for this change

Celeborn's mapStatus is wrong. #1072

Left side partitions size info:
median size: 1, max size: 0, min size: 0, avg size: 0
Right side partitions size info:
median size: 1, max size: 0, min size: 0, avg size: 0

PR test

25/08/08 20:45:05,334 [main] DEBUG OptimizeSkewedJoin:
Optimizing skewed join.
Left side partitions size info:
median size: 414921555, max size: 105074589412, min size: 413336727, avg size: 663447205
Right side partitions size info:
median size: 71065115, max size: 71416790, min size: 70698593, avg size: 71064333

25/08/08 20:45:05,348 [main] DEBUG OptimizeSkewedJoin: Left side partition 0 (97 GB) is skewed, split it into 252 parts.
25/08/08 20:45:05,349 [main] DEBUG OptimizeSkewedJoin: Left side partition 211 (19 GB) is skewed, split it into 51 parts.
25/08/08 20:45:05,350 [main] DEBUG OptimizeSkewedJoin: number of skewed partitions: left 2, right 0

What changes are included in this PR?

Are there any user-facing changes?

@cxzl25 cxzl25 marked this pull request as ready for review August 8, 2025 12:34
@turboFei
Copy link
Member

turboFei commented Aug 8, 2025

seems we can add celeborn integration testing in blaze in the future.

@turboFei
Copy link
Member

turboFei commented Aug 9, 2025

@merrily01
Copy link
Member

Looks like I missed updating Celeborn as well. Sorry about that, and thanks for catching it.

@richox richox merged commit 7a67504 into apache:master Aug 12, 2025
97 checks passed
turboFei added a commit to turboFei/blaze that referenced this pull request Aug 12, 2025
richox pushed a commit that referenced this pull request Aug 14, 2025
…ules (#1136)

* Extracting celeborn/uniffle/paimon code to seperated modules

* fix typo

* rename file

* Fix incorrect Celeborn mapStatus (#1133)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants