Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Make zombienet tests native friendly #6588

Merged
merged 7 commits into from
Jan 28, 2023
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions zombienet_tests/.set_env
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
pathprepend() {
for ((i=$#; i>0; i--));
do
ARG="$@[$i]"
if [ -d "$ARG" ] && [[ ":$PATH:" != *":$ARG:"* ]]; then
PATH="$ARG${PATH:+":$PATH"}"
fi
done
}

# paths are prepend in order, so you can manage with version will run
# by the order of this array
CUSTOM_PATHS=(
"<path>/polkadot/target/release"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in my case at least is my home directory. I think it make sense to change <path> here to ~ to have a default that can work without altering this file (at least in my case :P but I assume others have similar setups)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense, I will change it to have a nice default that can work out of the box.
Thanks!

"<path>/polkadot/target/testnet"
"<path>/cumulus/target/release"
)

pathprepend $CUSTOM_PATHS
export PATH=$PATH
43 changes: 43 additions & 0 deletions zombienet_tests/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,49 @@ _The content of this directory is meant to be used by Parity's private CI/CD inf
* [zombienet repo](https://github.com/paritytech/zombienet)
* [zombienet book](https://paritytech.github.io/zombienet/)

## Running tests locally

To run any test locally use the native provider (`zombienet test -p native ...`) you need first build the binaries. They are:

* adder-collator -> polkadot/target/testnet/adder-collator
* malus -> polkadot/target/testnet/malus
* polkadot -> polkadot/target/testnet/polkadot
* polkadot-collator -> cumulus/target/release/polkadot-parachain
* undying-collator -> polkadot/target/testnet/undying-collator

To build them use:
* adder-collator -> `cargo build --profile testnet -p test-parachain-adder-collator`
* undying-collator -> `cargo build --profile testnet -p test-parachain-undying-collator`
* malus -> cargo build --profile testnet -p polkadot-test-malus
* polkadot (in polkadot repo) and polkadot-collator (in cumulus repo) -> `cargo build --profile testnet`

One solution is to use the `.set_env` file (from this directory) and fill the `CUSTOM_PATHS` before *source* it to patch the PATH of your system to find the binaries you just built.

E.g.:
```
$ cat .set_env
(...)
# by the order of this array
CUSTOM_PATHS=(
"<path>/polkadot/target/release"
"<path>/polkadot/target/testnet"
"<path>/cumulus/target/release"
)
(...)

source .set_env
```

Then you have your `PATH` customized and ready to run `zombienet`.
**NOTE**: You should need to do this ones per terminal session, since we are patching the `PATH` and re-exporting.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can also source this env fine in .bashrc file for example which gets executed when you log in. So you don't have to do this every time.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, adding to .bashrc will be better and save some time.
Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We go in the personal preference zone :)
One might want to source it by hand, other - in bashrc, third (me :P ) - alias it and so on.

We've got a hint to use path, we've got a script to avoid useless copy-pasting/linking, we even have got sane-ish defaults. Let's merge as it is.


Example:

You can run a test locally by executing:
```sh
zombienet test -p native 0001-parachains-pvf.zndsl
```

## Questions / permissions

Ping in element Javier (@javier:matrix.parity.io) to ask questions or grant permission to run the test from your local setup.
6 changes: 3 additions & 3 deletions zombienet_tests/functional/0001-parachains-pvf.zndsl
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ eve: reports histogram polkadot_pvf_preparation_time has at least 1 samples in b
one: reports histogram polkadot_pvf_preparation_time has at least 1 samples in buckets ["0.1", "0.5", "1", "2", "3", "10"] within 10 seconds
two: reports histogram polkadot_pvf_preparation_time has at least 1 samples in buckets ["0.1", "0.5", "1", "2", "3", "10"] within 10 seconds

# Check all buckets >= 20.
# Check all buckets >= 20.
alice: reports histogram polkadot_pvf_preparation_time has 0 samples in buckets ["20", "30", "60", "180", "+Inf"] within 10 seconds
bob: reports histogram polkadot_pvf_preparation_time has 0 samples in buckets ["20", "30", "60", "180", "+Inf"] within 10 seconds
charlie: reports histogram polkadot_pvf_preparation_time has 0 samples in buckets ["20", "30", "60", "180", "+Inf"] within 10 seconds
Expand All @@ -78,8 +78,8 @@ two: reports histogram polkadot_pvf_preparation_time has 0 samples in buckets ["
# APPROVAL_EXECUTION_TIMEOUT(6s). Currently these are not differentiated by metrics
# because the metrics are defined in `polkadot-node-core-pvf` which is a level below
# the relevant subsystems.
# That being said, we will take the simplifying assumption of testing only the
# 2s timeout.
# That being said, we will take the simplifying assumption of testing only the
# 2s timeout.
# We do this check by ensuring all executions fall into bucket le="2" or lower.
# First, check if we have at least 1 sample, but we should have many more.
alice: reports histogram polkadot_pvf_execution_time has at least 1 samples in buckets ["0.1", "0.5", "1", "2"] within 10 seconds
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ requests = { memory = "2G", cpu = "1" }
command = "malus suggest-garbage-candidate"
args = ["-lparachain=debug,MALUS=trace"]
count = 1

{% for id in range(2000,2003) %}
[[parachains]]
id = {{id}}
Expand Down
2 changes: 1 addition & 1 deletion zombienet_tests/misc/0001-paritydb.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ requests = { memory = "2G", cpu = "1" }
name = "validator"
count = 10
args = ["-lparachain=debug", "--db=paritydb"]

{% for id in range(2000,2010) %}
[[parachains]]
id = {{id}}
Expand Down
2 changes: 1 addition & 1 deletion zombienet_tests/smoke/0001-parachains-smoke-test.toml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ addToGenesis = false
[parachains.collator]
name = "collator01"
image = "{{COL_IMAGE}}"
command = "/usr/local/bin/adder-collator"
command = "adder-collator"
args = [ "-lruntime=debug,parachain=trace" ]

[types.Header]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,3 @@ command = "polkadot"
[[relaychain.nodes]]
name = "dave"
args = [ "-lruntime=debug,parachain=trace" ]