Skip to content

Commit

Permalink
[#1572] fix(spark): Exceptions might be discarded when spilling buffe…
Browse files Browse the repository at this point in the history
…rs (#1573)

### What changes were proposed in this pull request?

Handle all the exceptions when spilling buffers.
Let users know when a TimeoutException occurs.

### Why are the changes needed?

Fix #1572.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Existing UTs.
  • Loading branch information
rickyma authored Mar 14, 2024
1 parent 9737d57 commit b2baef5
Showing 1 changed file with 3 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -482,6 +482,9 @@ public long spill(long size, MemoryConsumer trigger) {
} catch (TimeoutException timeoutException) {
// A best effort strategy to wait.
// If timeout exception occurs, the underlying tasks won't be cancelled.
LOG.warn("[taskId: {}] Spill tasks timeout after {} seconds", taskId, memorySpillTimeoutSec);
} catch (Exception e) {
LOG.warn("[taskId: {}] Failed to spill buffers due to ", taskId, e);
} finally {
long releasedSize =
futures.stream()
Expand Down

0 comments on commit b2baef5

Please sign in to comment.