feat(mocknet): add scripts to test mocknet itself locally

[marcelo-gonzalez]

This splits out common functionality previously written out in mirror.py into a NodeHandle class, and then provides two implementations, one for real mocknet tests over SSH, and one for local testing that starts neard runner processes locally. This is helpful because without this, it can be very difficult to test changes to the mocknet scripts themselves, as starting a real mocknet test can be expensive and time consuming

The local_test_node.py file added here includes a command that can be used to set up a local test with state taken from a local near home directory. An example of how to use it is:

$ cd pytest/
$ python3 tests/sanity/{your_favorite_test}
$ # find a height close to tail, trying different values until you get a DBNotFound error. Use a height close to the tail as the --fork-height argument in the next command
$ ~/nearcore/target/debug/neard --home ~/.near/test0_finished view-state view-chain --height {some_height}
$ python3 tests/mocknet/local_test_node.py local-test-setup --num-nodes 3 --source-home-dir ~/.near/test0_finished --neard-binary-path ~/nearcore/target/debug/neard --fork-height {some_height}
[2024-03-19 15:26:48] INFO: started neard runner process with pid 1489770 listening on port 3000
[2024-03-19 15:26:48] INFO: started neard runner process with pid 1489771 listening on port 3001
[2024-03-19 15:26:48] INFO: started neard runner process with pid 1489772 listening on port 3002
[2024-03-19 15:26:48] INFO: started neard runner process with pid 1489773 listening on port 3003
127.0.0.1 - - [19/Mar/2024 15:26:48] "GET /neard HTTP/1.1" 200 -
127.0.0.1 - - [19/Mar/2024 15:26:51] "GET /neard HTTP/1.1" 200 -
127.0.0.1 - - [19/Mar/2024 15:26:55] "GET /neard HTTP/1.1" 200 -
127.0.0.1 - - [19/Mar/2024 15:27:17] "GET /neard HTTP/1.1" 200 -
All directories initialized. neard runners are running in dirs: ['~/.near/local-mocknet/traffic-generator', '~/.near/local-mocknet/node0', '~/.near/local-mocknet/node1', '~/.near/local-mocknet/node2'], listening on respective ports: [3000, 3001, 3002, 3003]
$ # now use the normal mirror.py script with the --local-test flag instead of the usual --chain-id, --unique-id and --start-height
$ python3 tests/mocknet/mirror.py --local-test new-test
$ python3 tests/mocknet/mirror.py --local-test start-traffic

[marcelo-gonzalez]

@posvyatokum sorry it is a bit large. I tried to separate it into different commits to make it easier to review. It should all "just work" if everything is right, but note that it sometimes fails for me with neard built from master because of this: #10827. If you try it with 1.38.0 it should work. Also, I have indeed used this extensively on "real" mocknet, so it has been tested a good amount already for what it's worth

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 71.53%. Comparing base (d92c69d) to head (9e27bc1).

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #10835   +/-   ##
=======================================
  Coverage   71.53%   71.53%           
=======================================
  Files         758      758           
  Lines      151686   151686           
  Branches   151686   151686           
=======================================
+ Hits       108512   108515    +3     
+ Misses      38684    38682    -2     
+ Partials     4490     4489    -1     

Flag	Coverage Δ
backward-compatibility	0.24% <ø> (ø)
db-migration	0.24% <ø> (ø)
genesis-check	1.43% <ø> (ø)
integration-tests	37.09% <ø> (+0.02%)	⬆️
linux	70.00% <ø> (-0.01%)	⬇️
linux-nightly	71.01% <ø> (-0.01%)	⬇️
macos	54.48% <ø> (-0.01%)	⬇️
pytests	1.66% <ø> (ø)
sanity-checks	1.44% <ø> (ø)
unittests	67.18% <ø> (+<0.01%)	⬆️
upgradability	0.29% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[posvyatokum]

Love the idea, love the breakdown of mirror.py.
This ties to the idea of "Forknet=Nayduck with mocknet backend" by separating test orchestration and node handling.
Will say that in localnet pytests that we have in Nayduck this is done differently by having

class BaseNode
class RpcNode(BaseNode)
class LocalNode(BaseNode)

This is why I see another refactoring of this code in the future.