Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(indexer): Fix concurrency issues in ReindexThread (#30083) #30084

Merged

Conversation

spbolton
Copy link
Contributor

@spbolton spbolton commented Sep 20, 2024

Fixes concurrency issues and prevents race conditions that cause the indexer from hanging or shutting down immediately after startup.

This PR resolves #30083 (Startup timeouts due to indexer concurrency issues).

This PR fixes: #30083

@spbolton spbolton linked an issue Sep 20, 2024 that may be closed by this pull request
@spbolton spbolton changed the title fix(backend): Startup timeouts due to indexer concurrency issues (#30083) fix(indexer): Fix concurrency issues in ReindexThread (#30083) Sep 20, 2024
Copy link
Contributor

@dcolina dcolina left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏽

Copy link
Contributor

@wezell wezell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some comments - it is important to note that this is like touching a third rail.

I like a lot of these changes and where you are going with this Steve but my question to you would be what is the smallest PR that resolves the issue we are seeing now?

We will need to test a reindex in a cluster (3 node) with more content than the starter provides - something like 10000 pieces of content.

@wezell
Copy link
Contributor

wezell commented Sep 20, 2024

Important note - none of our ITs run against or expect a cluster to exist. This specifically needs to be tested by hand in a clustered configuration - 3 nodes - and test reindex starts, stops, switchovers, manual stops and manual switchovers.

@nollymar nollymar marked this pull request as draft September 20, 2024 22:46
@spbolton spbolton force-pushed the issue-30083-startup-timeouts-due-to-indexer-concurrency-issues branch 2 times, most recently from ce89f8b to c39c638 Compare September 23, 2024 10:14
@spbolton spbolton force-pushed the issue-30083-startup-timeouts-due-to-indexer-concurrency-issues branch from c39c638 to df44f60 Compare September 23, 2024 11:51
@spbolton
Copy link
Contributor Author

Modified this PR to a single change to hopefully fix the immediate issue and a follow up refactor work to fixup the general state management issues here #30083
#30110 as this will require more testing before it can be merged.

@spbolton spbolton marked this pull request as ready for review September 23, 2024 12:11
@spbolton
Copy link
Contributor Author

@wezell Can we push in just this change, I think it will help on its own with the one issue here and we can spend some more time of general concurrency and state management issues in the new Issue I have created.

@spbolton
Copy link
Contributor Author

Fixed sonarqube reported issue.
We were creating a "new Thread" that was not actually doing anything. We never started this thread object, it was working because thread itself extends Runnable so when passed to the Executor with submit it was just calling the run method of the "Runnable" we sent into the thread constructor. The thread name we set would not be used anywhere in the executor. We can just pass the original runnable getInstance().ReindexThreadRunnable directly into the submit method with the same effect. Creating the new Thread is both confusing and unecessary

Copy link
Contributor

@wezell wezell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great.

@nollymar nollymar added this pull request to the merge queue Sep 23, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 23, 2024
@spbolton spbolton added this pull request to the merge queue Sep 23, 2024
Merged via the queue into master with commit 462828c Sep 23, 2024
32 checks passed
@spbolton spbolton deleted the issue-30083-startup-timeouts-due-to-indexer-concurrency-issues branch September 23, 2024 22:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Startup timeouts due to indexer concurrency issues
4 participants