Add initial KV Tx RPC implementation #626

canepat · 2022-04-01T08:21:47Z

This PR introduces the initial implementation of KV Tx bidirectional RPC. Key points:

every new Tx incoming RPC creates a new MDBX read-only transaction
one unsolicited message containing the txn ID is notified to the client
cursor open/close and tracking by pairing db::Cursor with table name to support reopening
one max txn TTL timer protects against long-lived txn by aborting/reopening the txn on behalf of the client, opened cursors are saved/restored transparently
another idle timer protects against clients which don't send any (more) requests and eventually terminates the RPC

TODOLIST:

check errors when limits are reached (max number of txns, max number of cursors per txn), if applicable

Fix and add unit tests

Implement Tx RPC cursor operations [WIP] Fix cursor ID assignment in BackEndKV test client

codecov · 2022-04-01T21:27:45Z

Codecov Report

Merging #626 (d90a87e) into master (d57404a) will increase coverage by 0.27%.
The diff coverage is 87.27%.

@@            Coverage Diff             @@
##           master     #626      +/-   ##
==========================================
+ Coverage   82.09%   82.36%   +0.27%     
==========================================
  Files         173      173              
  Lines       14425    14997     +572     
==========================================
+ Hits        11842    12353     +511     
- Misses       2583     2644      +61

Impacted Files	Coverage Δ
node/silkworm/backend/ethereum_backend.cpp	`100.00% <ø> (ø)`
node/silkworm/rpc/client/sentry_client.hpp	`100.00% <ø> (ø)`
node/silkworm/rpc/completion_end_point.cpp	`100.00% <ø> (ø)`
node/silkworm/rpc/kv_calls.cpp	`87.39% <86.16%> (-1.27%)`	⬇️
node/silkworm/rpc/call.hpp	`93.23% <89.69%> (+2.75%)`	⬆️
node/silkworm/backend/ethereum_backend.hpp	`100.00% <100.00%> (ø)`
node/silkworm/rpc/backend_calls.cpp	`100.00% <100.00%> (ø)`
node/silkworm/rpc/backend_kv_server.cpp	`100.00% <100.00%> (ø)`
node/silkworm/rpc/call_factory.hpp	`100.00% <100.00%> (ø)`
node/silkworm/rpc/client/call.hpp	`67.85% <100.00%> (ø)`
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d57404a...d90a87e. Read the comment docs.

Refactor chaindata environment type

Handle any exception raised in mdbx::env::start_read Fix error handling in save and restore cursors Add MDBX max readers option in BackEnd KV server

Add unit test

Refactor for testability Add unit tests for coverage

Add unit tests for Tx max idle timeout

Add unit tests for max TTL duration Add unit tests for Tx cursor FIRST operation Add unit tests for Tx cursor NEXT operation

Fix cursor FIRST_DUP implementation in Tx RPC Fix cursor CURRENT implementation in Tx RPC Add unit tests

Workaround in FIRST_DUP for MDBX assert Fix logic in FIRST_DUP unit tests Add NEXT_DUP unit test

Add exception handling in save and restore

Refactor exception handling for cursor operations Add error for unhandled cursor operation Fix number of cursor positions Fix cursor position usage-after-destruction Add unit tests

node/silkworm/rpc/backend_calls.cpp

node/silkworm/rpc/kv_calls_test.cpp

node/silkworm/rpc/call.hpp

Improve naming

canepat added 2 commits April 1, 2022 09:48

Add KV Tx RPC implementation

ea2ca0a

Fix and add unit tests

Finish RPC with error when idle timer expires

cf188ae

Implement Tx RPC cursor operations [WIP] Fix cursor ID assignment in BackEndKV test client

canepat added 27 commits April 1, 2022 23:49

Add implementation for some Tx RPC cursor operations

171ab86

Add Tx RPC unit test for cursor FIRST

915dfae

Refactor chaindata environment type

Refactor BackEndKvServer unit tests

369837a

Add BackEndKvServer unit tests

56ede10

Try to fix gRPC data race

000ada5

Fix gRPC data race in unit tests: try 2

e432953

Fix gRPC data race in unit tests: try 3

71fa8df

Fix gRPC data race in unit tests: try 4

1920a41

Fix gRPC data race in unit tests: try 5

df07efe

Fix gRPC data race in unit tests: try 6

ac62703

Fix gRPC data race in unit tests: try 7

24a6225

Refactor bidi streaming RPC close to avoid double finish

c2347cc

Handle any exception raised in mdbx::env::start_read Fix error handling in save and restore cursors Add MDBX max readers option in BackEnd KV server

Change Sentry mock server from sync to async

0f0462b

Add missing KV Tx cursor operations

43598ce

Add unit test

Add unit tests

586650e

Remove unused server RPC method

bc8fe65

Refactor for testability Add unit tests for coverage

Configure max idle timeout for bidirectional RPC

655ce78

Add unit tests for Tx max idle timeout

Configure max TTL duration in Tx RPC

25b81bc

Add unit tests for max TTL duration Add unit tests for Tx cursor FIRST operation Add unit tests for Tx cursor NEXT operation

Add error handling for cursor operations in Tx RPC

cbedefb

Fix cursor FIRST_DUP implementation in Tx RPC Fix cursor CURRENT implementation in Tx RPC Add unit tests

Comment out FIRST_DUP unit tests

4dcf7f1

Comment out sleep_for in max TTL unit test

91db211

Print message to diagnose max TTL error on Windows

92ea7c6

Readd one FIRST_DUP unit test

e610c13

Print MDBX results for debugging

5458264

Hand-made print

264cf32

Change restore for single-value cursors in Tx RPC

b5c00f0

Add unit test

84875c9

canepat added 19 commits April 10, 2022 09:38

Add unit test

6ae13a3

Fix FIRST_DUP implementation

18cc285

Workaround in FIRST_DUP for MDBX assert Fix logic in FIRST_DUP unit tests Add NEXT_DUP unit test

Fix restore for DUPSORT cursor

b073fdd

Fix NEXT_DUP unit test

f77e452

Add exception handling in save and restore

Comment out unit test

57b97b1

Uncomment out unit test w/o sleep

0d389a3

Add trace log in restore

547a38c

Reinsert delay in max TTL unit test

a792429

Add print

ea14e63

Add PREV_DUP and PREV_NO_DUP cursor operations

faefad2

Refactor exception handling for cursor operations Add error for unhandled cursor operation Fix number of cursor positions Fix cursor position usage-after-destruction Add unit tests

Rebind existent cursors in restore

dfae185

Add unit test for exceeded max readers

e687b3c

Add unit test triggering server-side write error

9787a67

Add unit tests for CURRENT cursor operation

45859b5

Add unit tests

69c43f0

Comment out unit test

20acf74

Retry test for server-side write error

07b01ac

Add unit tests

8af28d0

Convert unfiltered logs to debug logs

cbb8cbf

canepat marked this pull request as ready for review April 13, 2022 20:10

canepat requested a review from mriccobene April 13, 2022 20:10

mriccobene approved these changes Apr 19, 2022

View reviewed changes

node/silkworm/rpc/backend_calls.cpp Show resolved Hide resolved

node/silkworm/rpc/kv_calls_test.cpp Outdated Show resolved Hide resolved

node/silkworm/rpc/call.hpp Show resolved Hide resolved

node/silkworm/rpc/call.hpp Show resolved Hide resolved

canepat added 2 commits April 19, 2022 17:32

Avoid unnecessary copies when sending responses

3eb6503

Improve naming

Merge branch 'master' into backend_kv_server6

d90a87e

canepat merged commit f4f6759 into master Apr 19, 2022

canepat deleted the backend_kv_server6 branch April 19, 2022 18:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial KV Tx RPC implementation #626

Add initial KV Tx RPC implementation #626

canepat commented Apr 1, 2022 •

edited

Loading

codecov bot commented Apr 1, 2022 •

edited

Loading

Add initial KV Tx RPC implementation #626

Add initial KV Tx RPC implementation #626

Conversation

canepat commented Apr 1, 2022 • edited Loading

codecov bot commented Apr 1, 2022 • edited Loading

Codecov Report

canepat commented Apr 1, 2022 •

edited

Loading

codecov bot commented Apr 1, 2022 •

edited

Loading