Repository navigation
Fix parallelism cap when current thread is enrolled in concurrent HNSW merge - #16778
Draft
abernardi597 wants to merge 3 commits into
Draft
abernardi597 wants to merge 3 commits into
abernardi597 wants to merge 3 commits into
Conversation
abernardi597
force-pushed
the
lucene-16729
branch
from
October 6, 2026 15:24
d50ee0e to
2d5755f
Compare
abernardi597
force-pushed
the
lucene-16729
branch
from
October 7, 2026 19:22
2d5755f to
fa0e2b1
Compare
abernardi597
force-pushed
the
lucene-16729
branch
from
October 7, 2026 19:26
fa0e2b1 to
f407da2
Compare
Contributor
|
Don't force push please. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #16729.
Description
HnswConcurrentMergeBuildersubmitted all of its workers throughTaskExecutor#invokeAllbefore any of them ran. The intra-merge executor runs a command on the calling thread when it has no spare thread, so when that happened to the first submission, the worker drained the shared batch counter before the submit loop could offer the remaining workers. Those workers then found no work left, and the merge stayed single-threaded for its whole duration even once sibling merges freed threads up.Change
Adds
TaskExecutor#invokeNoReentry, which forks as many tasks as the executor will accept and returns immediately, stopping as soon as the executor runs a command on the calling thread rather than letting that consume the caller.HnswConcurrentMergeBuildernow runs its first worker on the merge thread and offers the workers it has not placed yet again between batches, so intra-merge threads that become available mid-merge are picked up.TaskExecutor#invokeAllis untouched, so search, faceting and the BP reorderers are unaffected. Work distribution is still the shared counter, so dynamic balance across workers is unchanged.