feat(mocknet): add a make-backup command

[marcelo-gonzalez]

It is often useful to stop all the nodes and make backups of their data dirs so that we can reset to that point later. For example, when testing resharding it's nice to have a backup at a point right before the resharding epoch starts so that we can test the resharding part over and over without having to start from the beginning. So this PR adds a make_backup RPC method to neard_runner.py that stops neard and copies its data dir, remembering the backup_id parameter associated with it. Then in the reset command, the same backup_id can be referenced to restore to a previous backup.

[marcelo-gonzalez]

@wacban after #10835 it is now possible to test this PR locally. You can do it like this:

run some pytest. I tried it with transactions.py edited like so:

diff --git a/pytest/tests/sanity/transactions.py b/pytest/tests/sanity/transactions.py
index c628fbee0..2dd9b286e 100755
--- a/pytest/tests/sanity/transactions.py
+++ b/pytest/tests/sanity/transactions.py
@@ -23,7 +23,7 @@ nodes = start_cluster(
     num_shards=4,
     config=None,
     genesis_config_changes=[["min_gas_price",
-                             0], ["max_inflation_rate", [0, 1]],
+                             0], ["use_production_config", True], ["max_inflation_rate", [0, 1]],
                             ["epoch_length", 10],
                             ["block_producer_kickout_threshold", 70]],
     client_config_changes={
@@ -33,7 +33,8 @@ nodes = start_cluster(
                     "secs": 2,
                     "nanos": 0
                 }
-            }
+            },
+            "tracked_shards": [0, 1, 2, 3]
         },
         1: {
             "consensus": {
@@ -41,7 +42,8 @@ nodes = start_cluster(
                     "secs": 2,
                     "nanos": 0
                 }
-            }
+            },
+            "tracked_shards": [0, 1, 2, 3]
         },
         2: {
             "consensus": {
@@ -49,7 +51,8 @@ nodes = start_cluster(
                     "secs": 2,
                     "nanos": 0
                 }
-            }
+            },
+            "tracked_shards": [0, 1, 2, 3]
         },
         3: {
             "consensus": {
@@ -57,7 +60,8 @@ nodes = start_cluster(
                     "secs": 2,
                     "nanos": 0
                 }
-            }
+            },
+            "tracked_shards": [0, 1, 2, 3]
         },
         4: {
             "consensus": {

then run:

$ python3 tests/mocknet/local_test_node.py local-test-setup --yes --num-nodes 2 --source-home-dir ~/.near/test4_finished/ --neard-binary-path ~/nearcore/target/debug/neard --fork-height 70
[2024-04-03 18:52:27] INFO: started neard runner process with pid 2674847 listening on port 3000
[2024-04-03 18:52:27] INFO: started neard runner process with pid 2674848 listening on port 3001
[2024-04-03 18:52:27] INFO: started neard runner process with pid 2674849 listening on port 3002
127.0.0.1 - - [03/Apr/2024 18:52:27] "GET /neard HTTP/1.1" 200 -
127.0.0.1 - - [03/Apr/2024 18:52:29] "GET /neard HTTP/1.1" 200 -
127.0.0.1 - - [03/Apr/2024 18:52:32] "GET /neard HTTP/1.1" 200 -
All directories initialized. neard runners are running in dirs: ['~/.near/local-mocknet/traffic-generator', '~/.near/local-mocknet/node0', '~/.near/local-mocknet/node1'], listening on respective ports: [3000, 3001, 3002]

$ python3 tests/mocknet/mirror.py --local-test new-test
$ python3 tests/mocknet/mirror.py --local-test start-traffic
# wait a bit
$ python3 tests/mocknet/mirror.py --local-test make-backup
$ python3 tests/mocknet/mirror.py --local-test start-traffic
# wait a bit
$ python3 tests/mocknet/mirror.py --local-test reset
# repeat make-backup / reset as you like
# then when you're done:
$ python3 tests/mocknet/mirror.py --local-test stop-nodes
$ python3 tests/mocknet/mirror.py --local-test stop-neard-runner

[marcelo-gonzalez]

also just occurred to me that we might as well make checkpoints w hard links instead of copying everything... could do that in a separate PR after this one though

[wacban]

That is fantastic thanks!

• It would also be cool to be able to specify upfront at what block height / epoch height we want the backup to be taken. Just an idea for another PR if you also think it would be useful :)
• If I recall correctly the data and home directory are actually mounted separately so there won't be any difference between copy and hard links.

[wacban]

nit: You can consider addding 'start' to the backups in self.data['backups'] when start is taken, then you won't need to special case this.

[marcelo-gonzalez]

ah actually i did include that, but i wanted to add this special case here so that this will work if you start neard_runner.py that includes this PR on a directory that was made by an older version, so it will "just work" in that case. same kind of thing behind backup_id being an optional arg of this RPC method, so that an older mirror.py will work with a newer neard_runner.py running on the mocknet nodes

[wacban]

This check should happen both for backup_id provided in args and in readline (so outside this if)

[marcelo-gonzalez]

mm well the reason i left it out when passing the backup ID on the command line was that when you dont pass it on the command line, we want to call ls-backups to get a list of backups to show the user. but when you do pass it on the command line, we can just skip that extra round trip and send it anyway, and neard_runner.py will just check that it doesnt exist. so you get the same behavior where neard_runner.py tells you it doesnt exist, but with only one RPC call

[marcelo-gonzalez]

• If I recall correctly the data and home directory are actually mounted separately so there won't be any difference between copy and hard links.

Yea true, but in this PR I moved the neard-runner dir to be inside ~/.near/, because even forgetting about checkpoints with hard links, there's often not even enough space to put the backups on the other disk, so in the future might make sense to change this to make cheaper checkpoints

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.54%. Comparing base (efd0b83) to head (0d329a1).
Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10931      +/-   ##
==========================================
+ Coverage   71.51%   71.54%   +0.02%     
==========================================
  Files         758      758              
  Lines      151663   152058     +395     
  Branches   151663   152058     +395     
==========================================
+ Hits       108467   108794     +327     
- Misses      38710    38761      +51     
- Partials     4486     4503      +17     

Flag	Coverage Δ
backward-compatibility	0.24% <ø> (-0.01%)	⬇️
db-migration	0.24% <ø> (-0.01%)	⬇️
genesis-check	1.42% <ø> (-0.01%)	⬇️
integration-tests	37.16% <ø> (+0.04%)	⬆️
linux	70.02% <ø> (+0.04%)	⬆️
linux-nightly	71.02% <ø> (+0.01%)	⬆️
macos	54.52% <ø> (+0.12%)	⬆️
pytests	1.65% <ø> (-0.01%)	⬇️
sanity-checks	1.44% <ø> (-0.01%)	⬇️
unittests	67.19% <ø> (+0.02%)	⬆️
upgradability	0.29% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.