Fix the lock order issue caused by value operator init #9738
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
facebook-github-bot
added
the
CLA Signed
✅ Deploy Preview for meta-velox canceled.
|
|
@xiaoxmeng has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
xiaoxmeng
added a commit
to xiaoxmeng/velox
that referenced
this pull request
May 7, 2024
…ator#9738) Summary: The lock order issue detected by tsan is as follows: T1: task startup creates driver and constructor operators which holds the task lock T2: value operator constructor allocates memory from memory pool for parallelized value node mode which triggers memory arbitration T3: the memory arbitration grabs arbitration lock for arbitration T4: the memory reclaim tries to grabs the task lock for reclaim such as to pause the task execution This forms a deadlock and we have the logic to detect the memory pool allocation from the operator constructor in driver execution framework. However the memory pool of vectors used by value nodes is not the query memory pool but owns by the test framework. This PR moves the memory pool allocation from constructor to initialize() method as the other operator does. Test Plan: The test passed 100 stress test iterations: https://www.internalfb.com/intern/testinfra/testrun/10414574171435627 https://www.internalfb.com/intern/testinfra/testrun/13229323938244819 Differential Revision: D57063553 Pulled By: xiaoxmeng
|
This pull request was exported from Phabricator. Differential Revision: D57063553 |
xiaoxmeng
added a commit
to xiaoxmeng/velox
that referenced
this pull request
May 7, 2024
…ator#9738) Summary: The lock order issue detected by tsan is as follows: T1: task startup creates driver and constructor operators which holds the task lock T2: value operator constructor allocates memory from memory pool for parallelized value node mode which triggers memory arbitration T3: the memory arbitration grabs arbitration lock for arbitration T4: the memory reclaim tries to grabs the task lock for reclaim such as to pause the task execution This forms a deadlock and we have the logic to detect the memory pool allocation from the operator constructor in driver execution framework. However the memory pool of vectors used by value nodes is not the query memory pool but owns by the test framework. This PR moves the memory pool allocation from constructor to initialize() method as the other operator does. Test Plan: The test passed 100 stress test iterations: https://www.internalfb.com/intern/testinfra/testrun/10414574171435627 https://www.internalfb.com/intern/testinfra/testrun/13229323938244819 Differential Revision: D57063553 Pulled By: xiaoxmeng
|
This pull request was exported from Phabricator. Differential Revision: D57063553 |
bikramSingh91
approved these changes
May 7, 2024
…ator#9738) Summary: The lock order issue detected by tsan is as follows: T1: task startup creates driver and constructor operators which holds the task lock T2: value operator constructor allocates memory from memory pool for parallelized value node mode which triggers memory arbitration T3: the memory arbitration grabs arbitration lock for arbitration T4: the memory reclaim tries to grabs the task lock for reclaim such as to pause the task execution This forms a deadlock and we have the logic to detect the memory pool allocation from the operator constructor in driver execution framework. However the memory pool of vectors used by value nodes is not the query memory pool but owns by the test framework. This PR moves the memory pool allocation from constructor to initialize() method as the other operator does. Test Plan: The test passed 100 stress test iterations: https://www.internalfb.com/intern/testinfra/testrun/10414574171435627 https://www.internalfb.com/intern/testinfra/testrun/13229323938244819 Reviewed By: bikramSingh91 Differential Revision: D57063553 Pulled By: xiaoxmeng
|
This pull request was exported from Phabricator. Differential Revision: D57063553 |
|
@xiaoxmeng merged this pull request in ccbb72e. |
|
Conbench analyzed the 1 benchmark run on commit There was 1 benchmark result indicating a performance regression:
The full Conbench report has more details. |
Joe-Abraham
pushed a commit
to Joe-Abraham/velox
that referenced
this pull request
May 8, 2024
…ator#9738) Summary: The lock order issue detected by tsan is as follows: T1: task startup creates driver and constructor operators which holds the task lock T2: value operator constructor allocates memory from memory pool for parallelized value node mode which triggers memory arbitration T3: the memory arbitration grabs arbitration lock for arbitration T4: the memory reclaim tries to grabs the task lock for reclaim such as to pause the task execution This forms a deadlock and we have the logic to detect the memory pool allocation from the operator constructor in driver execution framework. However the memory pool of vectors used by value nodes is not the query memory pool but owns by the test framework. This PR moves the memory pool allocation from constructor to initialize() method as the other operator does. Pull Request resolved: facebookincubator#9738 Test Plan: The test passed 100 stress test iterations: https://www.internalfb.com/intern/testinfra/testrun/10414574171435627 https://www.internalfb.com/intern/testinfra/testrun/13229323938244819 Reviewed By: bikramSingh91 Differential Revision: D57063553 Pulled By: xiaoxmeng fbshipit-source-id: 3ad661c35bfa46693729e60420f22920f83a7ff0
Joe-Abraham
pushed a commit
to Joe-Abraham/velox
that referenced
this pull request
Jun 7, 2024
…ator#9738) Summary: The lock order issue detected by tsan is as follows: T1: task startup creates driver and constructor operators which holds the task lock T2: value operator constructor allocates memory from memory pool for parallelized value node mode which triggers memory arbitration T3: the memory arbitration grabs arbitration lock for arbitration T4: the memory reclaim tries to grabs the task lock for reclaim such as to pause the task execution This forms a deadlock and we have the logic to detect the memory pool allocation from the operator constructor in driver execution framework. However the memory pool of vectors used by value nodes is not the query memory pool but owns by the test framework. This PR moves the memory pool allocation from constructor to initialize() method as the other operator does. Pull Request resolved: facebookincubator#9738 Test Plan: The test passed 100 stress test iterations: https://www.internalfb.com/intern/testinfra/testrun/10414574171435627 https://www.internalfb.com/intern/testinfra/testrun/13229323938244819 Reviewed By: bikramSingh91 Differential Revision: D57063553 Pulled By: xiaoxmeng fbshipit-source-id: 3ad661c35bfa46693729e60420f22920f83a7ff0
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The lock order issue detected by tsan is as follows:
T1: task startup creates driver and constructor operators which holds the task lock
T2: value operator constructor allocates memory from memory pool for parallelized value node mode
which triggers memory arbitration
T3: the memory arbitration grabs arbitration lock for arbitration
T4: the memory reclaim tries to grabs the task lock for reclaim such as to pause the task execution
This forms a deadlock and we have the logic to detect the memory pool allocation from the operator
constructor in driver execution framework. However the memory pool of vectors used by value nodes
is not the query memory pool but owns by the test framework. This PR moves the memory pool allocation
from constructor to initialize() method as the other operator does.