Fix deadlock in IO thread shutdown during panic#2898
Fix deadlock in IO thread shutdown during panic#2898ranshid merged 1 commit intovalkey-io:unstablefrom
Conversation
1be7b94 to
af34ff0
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## unstable #2898 +/- ##
=========================================
Coverage 72.45% 72.46%
=========================================
Files 129 129
Lines 70526 70528 +2
=========================================
+ Hits 51102 51109 +7
+ Misses 19424 19419 -5
🚀 New features to boost your workflow:
|
The IO thread shutdown could deadlock when the main thread calls pthread_cancel() while the IO thread holds its mutex. The IO thread would not observe the cancellation request, causing pthread_join() to block indefinitely. Release the IO thread mutex before cancelling to ensure the thread can process the cancellation request and exit cleanly. Reproducer: ./src/valkey-server --io-threads 2 --enable-debug-command yes ./src/valkey-cli debug panic Signed-off-by: Ouri Half <ourih@amazon.com>
af34ff0 to
8ba08f8
Compare
|
This deadlock was introduced by commit 3631208 which removed When This creates a deadlock scenario during panic shutdown:
The fix ensures the mutex is released before cancellation, allowing the thread to process the cancellation request in its main loop where |
## Problem IO thread shutdown can deadlock during server panic when the main thread calls `pthread_cancel()` while the IO thread holds its mutex, preventing the thread from observing the cancellation. ## Solution Release the IO thread mutex before cancelling to ensure clean thread termination. ## Testing Reproducer: ``` bash ./src/valkey-server --io-threads 2 --enable-debug-command yes ./src/valkey-cli debug panic ``` Before: Server hangs indefinitely After: Server terminates cleanly Signed-off-by: Ouri Half <ourih@amazon.com>
## Problem IO thread shutdown can deadlock during server panic when the main thread calls `pthread_cancel()` while the IO thread holds its mutex, preventing the thread from observing the cancellation. ## Solution Release the IO thread mutex before cancelling to ensure clean thread termination. ## Testing Reproducer: ``` bash ./src/valkey-server --io-threads 2 --enable-debug-command yes ./src/valkey-cli debug panic ``` Before: Server hangs indefinitely After: Server terminates cleanly Signed-off-by: Ouri Half <ourih@amazon.com>
## Problem IO thread shutdown can deadlock during server panic when the main thread calls `pthread_cancel()` while the IO thread holds its mutex, preventing the thread from observing the cancellation. ## Solution Release the IO thread mutex before cancelling to ensure clean thread termination. ## Testing Reproducer: ``` bash ./src/valkey-server --io-threads 2 --enable-debug-command yes ./src/valkey-cli debug panic ``` Before: Server hangs indefinitely After: Server terminates cleanly Signed-off-by: Ouri Half <ourih@amazon.com>
## Problem IO thread shutdown can deadlock during server panic when the main thread calls `pthread_cancel()` while the IO thread holds its mutex, preventing the thread from observing the cancellation. ## Solution Release the IO thread mutex before cancelling to ensure clean thread termination. ## Testing Reproducer: ``` bash ./src/valkey-server --io-threads 2 --enable-debug-command yes ./src/valkey-cli debug panic ``` Before: Server hangs indefinitely After: Server terminates cleanly Signed-off-by: Ouri Half <ourih@amazon.com>
Problem
IO thread shutdown can deadlock during server panic when the main thread calls
pthread_cancel()while the IO thread holds its mutex, preventing the thread from observing the cancellation.Solution
Release the IO thread mutex before cancelling to ensure clean thread termination.
Testing
Reproducer:
Before: Server hangs indefinitely
After: Server terminates cleanly