-
Notifications
You must be signed in to change notification settings - Fork 61
Introduce the Env::nested_read_txn
from an RwTxn
#307
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Kerollmops
wants to merge
12
commits into
main
Choose a base branch
from
allow-nested-rtxn-from-wtxn
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
95d39b5
to
f7651f0
Compare
This was referenced Jan 27, 2025
Kerollmops
commented
Feb 26, 2025
9a94c8a
to
1d1926b
Compare
1a8a21c
to
b7ebf74
Compare
ab18780
to
0c6793c
Compare
bdf9c69
to
81ae19f
Compare
5fcc354
to
a54defd
Compare
a54defd
to
05d9f22
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is a first attempt at patching LMDB to generate multiple
RoTxn
from anRwTxn
and, therefore, be able to read the content of an uncommittedRwTxn
in parallel. We can run many algorithms in Meilisearch to process the written content and generate new data structures before committing the transaction.This first attempt uses a broken fork of LMDB that ignores the number of nested write transactions associated with the parent one. Therefore, it lets heed create multiple nested write transactions from the parent one (dangerous). We store those write transactions in
RoTxn
s shells to disallow writing anything with them at the compilation level.However, while thinking about this, it is probably still possible to open databases (already existing) and commit them from multiple threads concurrently. It should be either disallowed or will, anyway, be fixed by the correct LMDB patch later.
To fix the Windows CI we will have to use the experimentalActually, no. We had to enable thestdatomic
headers./std:c11
and/experimental:c11atomics
features.Note that the CI is broken, but it works locally, so it's maybe a submodule issue (it's always a submodule issue) 🤔It was...To do
nested-rtxns
example in the CI (build it in release)