Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add lock to avoid segfault while resizing #61

Merged
merged 18 commits into from
Mar 22, 2024
Merged

Conversation

dylanrb123
Copy link
Contributor

@dylanrb123 dylanrb123 commented Mar 20, 2024

There is some unsafe memory access in the resizeIndex method, where it deletes and moves around memory while other threads may be accessing it. This PR adds a read-write lock to ensure that the resizeIndex function has exclusive access to the underlying resources when it needs to move the memory around

@dylanrb123 dylanrb123 requested a review from psobot March 20, 2024 21:25
@dylanrb123 dylanrb123 changed the title Add lock avoid segfault while resizing Add lock to avoid segfault while resizing Mar 21, 2024
Copy link
Member

@psobot psobot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also need to take a shared lock over resizeLock when calling addPoint? (addPoint is reading the data structures that get mutated while calling resizeIndex, and IIRC would also hit a segfault if we call addItems in parallel with resizeIndex.)

cpp/hnswalg.h Outdated Show resolved Hide resolved
cpp/hnswalg.h Outdated Show resolved Hide resolved
cpp/hnswalg.h Outdated Show resolved Hide resolved
cpp/hnswalg.h Outdated Show resolved Hide resolved
cpp/hnswalg.h Outdated Show resolved Hide resolved
@dylanrb123
Copy link
Contributor Author

Do we also need to take a shared lock over resizeLock when calling addPoint? (addPoint is reading the data structures that get mutated while calling resizeIndex, and IIRC would also hit a segfault if we call addItems in parallel with resizeIndex.)

addPoint only gets called after resizeIndex returns in the addItems function of TypedIndex so that wouldn't be possible in normal operation, however if someone is manually calling resizeIndex in one thread while calling addItems in another then it could cause trouble. I could take the shared lock just in case but seems pretty unlikely to happen in practice, what do you think?

@psobot
Copy link
Member

psobot commented Mar 21, 2024

I think it's still definitely worth locking there too - there are cases where multiple concurrent calls to addItems could still cause a segfault here; i.e.:

Thread A Thread B
Call addItems
Call addItems
Call addPoint
cur_element_count++
Index full, call resizeIndex
data_level0_memory_new = realloc(data_level0_memory_, ...)
*data_level0_memory_ 💥

@dylanrb123
Copy link
Contributor Author

👍 sounds good, will try to replicate in test and add the lock

@dylanrb123 dylanrb123 merged commit cc02679 into main Mar 22, 2024
53 checks passed
@dylanrb123 dylanrb123 deleted the index-while-resizing branch March 22, 2024 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants