Skip to content

Commit

Permalink
[CELEBORN-775][FOLLOWUP] Fix executorCores calculation in SparkShuffl…
Browse files Browse the repository at this point in the history
…eManager for Spark2 local mode

### What changes were proposed in this pull request?

### Why are the changes needed?
CELEBORN-775 The `executorCores` method was introduced in Spark2, but it was not used.

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
GA

Closes apache#2562 from cxzl25/CELEBORN-775-FOLLOWUP.

Authored-by: sychen <sychen@ctrip.com>
Signed-off-by: SteNicholas <programgeek@163.com>
  • Loading branch information
cxzl25 authored and SteNicholas committed Jun 13, 2024
1 parent cb30e91 commit e177a20
Showing 1 changed file with 1 addition and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ public SparkShuffleManager(SparkConf conf, boolean isDriver) {
this.conf = conf;
this.isDriver = isDriver;
this.celebornConf = SparkUtils.fromSparkConf(conf);
this.cores = conf.getInt(SparkLauncher.EXECUTOR_CORES, 1);
this.cores = executorCores(conf);
this.fallbackPolicyRunner = new CelebornShuffleFallbackPolicyRunner(celebornConf);
this.sendBufferPoolCheckInterval = celebornConf.clientPushSendBufferPoolExpireCheckInterval();
this.sendBufferPoolExpireTimeout = celebornConf.clientPushSendBufferPoolExpireTimeout();
Expand Down

0 comments on commit e177a20

Please sign in to comment.