Skip to content

Support Celeborn 0.5.x version#1124

Closed
cxzl25 wants to merge 1 commit intoapache:masterfrom
cxzl25:celeborn_0.5.x
Closed

Support Celeborn 0.5.x version#1124
cxzl25 wants to merge 1 commit intoapache:masterfrom
cxzl25:celeborn_0.5.x

Conversation

@cxzl25
Copy link
Contributor

@cxzl25 cxzl25 commented Aug 7, 2025

Which issue does this PR close?

Closes #.

Rationale for this change

Now jar has requirements for the Celeborn version, and users need to upgrade the Celeborn version, or revert related commit.

Caused by: java.lang.NoSuchMethodError: org.apache.spark.shuffle.celeborn.CelebornShuffleReader.<init>(Lorg/apache/spark/shuffle/celeborn/CelebornShuffleHandle;IIIILorg/apache/spark/TaskContext;Lorg/apache/celeborn/common/CelebornConf;Lorg/apache/spark/shuffle/ShuffleReadMetricsReporter;Lorg/apache/spark/shuffle/celeborn/ExecutorShuffleIdTracker;Z)V
	at org.apache.spark.sql.execution.blaze.shuffle.celeborn.BlazeCelebornShuffleReader$$anon$4.<init>(BlazeCelebornShuffleReader.scala:112)
	at org.apache.spark.sql.execution.blaze.shuffle.celeborn.BlazeCelebornShuffleReader.readBlocks(BlazeCelebornShuffleReader.scala:102)
	at org.apache.spark.sql.execution.blaze.shuffle.BlazeBlockStoreShuffleReaderBase.readIpc(BlazeBlockStoreShuffleReaderBase.scala:38)

What changes are included in this PR?

Are there any user-facing changes?

@cxzl25 cxzl25 marked this pull request as ready for review August 7, 2025 11:27
@richox
Copy link
Contributor

richox commented Aug 8, 2025

i'm thinking about extracting these 3rdparty components (celeborn, uniffle, paimon etc) to separated packages, then users can build with specified components for their environments, like build/mvn -Prelease -Pspark-3.5 -Pceleborn-0.6

@richox
Copy link
Contributor

richox commented Aug 15, 2025

close this issue as we have merged #1136

@richox richox closed this Aug 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants