[CORE-2698] Transaction log store #149
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
📊 Benchmark Resultsencoding.bench.tsKey encoding > ordered-binary keys - strings (100 records)
Key encoding > ordered-binary keys - numbers (100 records)
Key encoding > ordered-binary keys - mixed types (100 records)
Value encoding > msgpack values - strings (100 records)
Value encoding > msgpack values - numbers (100 records)
Value encoding > msgpack values - arrays (100 records)
Value encoding > msgpack values - small objects (100 records)
Value encoding > msgpack values - large objects (100 records)
get-sync.bench.tsgetSync() > random keys - small key size (100 records)
get.bench.tsget() > rocksdb - random vs sequential keys (100 records)
get() > random keys - max 1978 lmdb key size (100 records)
get() > rocksdb - async vs sync
put-sync.bench.tsputSync() > random keys - insert - small key size (100 records)
putSync() > random keys - update - small key size (100 records)
putSync() > random keys - insert - max 1978 lmdb key size (100 records)
putSync() > random keys - update - max 1978 lmdb key size (100 records)
putSync() > sequential keys - insert (100 records)
putSync() > put 100KB value (100 records)
putSync() > put 1MB value (100 records)
putSync() > get 10MB value (100 records)
put.bench.tsput > small dataset (100 records)
put > async vs sync overhead
ranges.bench.tsgetRange() > small range (100 records, 50 range)
getRange() > full scan vs range scan
getKeys() > keys only (100 records, 50 range)
Reverse iteration > reverse range (100 records, 50 range)
Reverse iteration > rocksdb - reverse vs forward
Range query patterns > prefix scan performance
Sparse data patterns > sparse - range over gaps
Sparse data patterns > sparse - prefix with gaps
remove-sync.bench.tsremoveSync() > random keys - small key size (100 records)
removeSync() > sequential keys - small key size (100 records)
removeSync() > rocksdb - random vs sequential keys (100 records)
removeSync() > random keys - max 1978 lmdb key size (100 records)
removeSync() > random access pattern (100 records)
removeSync() > non-existent keys (100 records)
transaction-sync.bench.tstransaction sync > optimistic > simple put operations (100 records)
transaction sync > optimistic > batch operations (100 records)
transaction sync > optimistic > read-write operations (100 records)
transaction sync > optimistic > concurrent non-conflicting operations (100 records)
transaction sync > optimistic > rollback operations (100 records)
transaction sync > optimistic > rocksdb - large transaction vs many small
transaction sync > optimistic > lmdb - large transaction vs many small
transaction sync > optimistic > empty transaction overhead
transaction sync > optimistic > transaction with only reads (100 records)
transaction sync > pessimistic > simple put operations (100 records)
transaction.bench.tstransaction > optimistic > simple put operations (100 records)
transaction > optimistic > batch operations (100 records)
transaction > optimistic > read-write operations (100 records)
transaction > optimistic > concurrent non-conflicting operations (100 records)
transaction > optimistic > rollback operations (100 records)
transaction > optimistic > rocksdb - large transaction vs many small
transaction > optimistic > lmdb - large transaction vs many small
transaction > optimistic > empty transaction overhead
transaction > optimistic > transaction with only reads (100 records)
transaction > pessimistic > simple put operations (100 records)
worker-get-sync.bench.tsWorker > random keys - small key size (100 records, 1 worker)
Worker > random keys - small key size (100 records, 2 workers)
Worker > random keys - small key size (100 records, 10 workers)
worker-put-sync.bench.tsputSync() > random keys - small key size (100 records, 1 worker)
putSync() > random keys - small key size (100 records, 2 workers)
putSync() > random keys - small key size (100 records, 10 workers)
Results from commit c87e14a |
kriszyp
reviewed
Oct 27, 2025
src/binding/transaction_log.cpp
Outdated
|
|
||
| // // create a reference to pin the buffer in memory (prevents GC) | ||
| napi_ref bufferRef = nullptr; | ||
| NAPI_STATUS_THROWS(::napi_create_reference(env, argv[0], 1, &bufferRef)); |
There was a problem hiding this comment.
I think we should copy the contents of the buffer into data, keep this as managed memory in C (delete when done) rather than maintaining a JS ref:
- Remove the complexity of managing, tracking, and deleting the JS reference
- Allow the JS buffer to be immediately GC'ed. For very large txns, helps protect against JS OOM, better scalability.
- Or allow for the safe reuse of shared buffer, that can be written over and over. Right now, I think this expects that the buffer won't be modified until the commit takes place, but this shouldn't be assumed, copying the data keeps it safe.
- Minimizes napi calls/interactions with the buffer (which is generally expensive, I believe).
… we find a valid log file to write to
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIP