Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Make zombienet tests native friendly #6588

Merged
merged 7 commits into from
Jan 28, 2023
Merged

Make zombienet tests native friendly #6588

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
931b4dc
Make zombienet tests native friendly
tdimitrov Jan 19, 2023
523e2c6
Update zombienet_tests/README.md
tdimitrov Jan 21, 2023
7d182d2
Get rid of `{{ZOMBIENET_NATIVE_BINARY_PREFIX}}`
tdimitrov Jan 21, 2023
a563ac6
Remove pointless changes
tdimitrov Jan 23, 2023
71006a9
add set_env helper and doc in readme
pepoviola Jan 25, 2023
eaac881
changes from feedback
pepoviola Jan 26, 2023
5a981da
Merge branch 'master' into tsv-zombienet-native
tdimitrov Jan 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions zombienet_tests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,35 @@ _The content of this directory is meant to be used by Parity's private CI/CD inf
* [zombienet repo](https://github.com/paritytech/zombienet)
* [zombienet book](https://paritytech.github.io/zombienet/)

## Running tests locally

To run any test locally use the native provider (`zombienet test -p native ...`) and set `ZOMBIENET_NATIVE_BINARY_PREFIX` environment variable with a full path to a directory containing polkadot/cumulus binaries. E.g. `ZOMBIENET_NATIVE_BINARY_PREFIX=/home/user/local-run/`. **The path should contain a trailing slash!**
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we not remove the trailing slash?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(By changing the zombienet tests?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually we can do this in a much simpler way. Instead of hardcoding the path in the test file we can just modify the PATH variable for the execution. E.g.:

$ PATH=/home/user/local-run/:$PATH zombienet test -p native 0001-parachains-pvf.zndsl

If I'm not mistaken, binaries are looked up in each directory, in the order they are listed in PATH, so it will work just fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I worded the last sentence in an awful way but you understand what I mean.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably depends on how zombienet is implemented? But if you already have some custom env variable, why trying to build in some second way? And if we follow your thinking, we could remove the custom env variable directly and people could modify the PATH only.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually this is what I'm suggesting - I removed ZOMBIENET_NATIVE_BINARY_PREFIX altogether.

But if you already have some custom env variable, why trying to build in some second way?

I don't understand this. What do you mean with 'build in some second way'?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignore what I said. 🙈 I have now seen your latest changes! Looking good. 👍


You need the following binaries:
* adder-collator -> polkadot/target/testnet/adder-collator
* malus -> polkadot/target/testnet/malus
* polkadot -> polkadot/target/testnet/polkadot
* polkadot-collator -> cumulus/target/release/polkadot-parachain
* undying-collator -> polkadot/target/testnet/undying-collator

To build them use:
* adder-collator -> `cargo build --profile testnet -p test-parachain-adder-collator`
* undying-collator -> `cargo build --profile testnet -p test-parachain-undying-collator`
* malus -> cargo build --profile testnet -p polkadot-test-malus
* polkadot (in polkadot repo) and polkadot-collator (in cumulus repo) -> `cargo build --profile testnet`

One solution is to have a directory with symlinks to the corresponding binaries in your source tree. E.g.:
```
$ ls -l
total 24
lrwxrwxrwx. 1 ceco ceco 53 Jan 19 15:03 adder-collator -> /home/ceco/src/polkadot/target/testnet/adder-collator
-rw-r--r--. 1 ceco ceco 221 Jan 18 16:35 build_cmds.txt
lrwxrwxrwx. 1 ceco ceco 44 Jan 18 16:47 malus -> /home/user/src/polkadot/target/testnet/malus
lrwxrwxrwx. 1 ceco ceco 47 Jan 18 11:33 polkadot -> /home/user/src/polkadot/target/testnet/polkadot
lrwxrwxrwx. 1 ceco ceco 56 Jan 19 15:53 polkadot-collator -> /home/user/src/cumulus/target/release/polkadot-parachain
lrwxrwxrwx. 1 ceco ceco 55 Jan 18 13:58 undying-collator -> /home/user/src/polkadot/target/testnet/undying-collator
```
And set this directory in `ZOMBIENET_NATIVE_BINARY_PREFIX`. This way you won't have to copy files on each rebuild.
tdimitrov marked this conversation as resolved.
Show resolved Hide resolved
## Questions / permissions

Ping in element Javier (@javier:matrix.parity.io) to ask questions or grant permission to run the test from your local setup.
35 changes: 18 additions & 17 deletions zombienet_tests/functional/0001-parachains-pvf.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,9 @@ timeout = 1000

[relaychain]
default_image = "{{ZOMBIENET_INTEGRATION_TEST_IMAGE}}"
default_command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}polkadot"
chain = "rococo-local"
chain_spec_command = "polkadot build-spec --chain rococo-local --disable-default-bootnode"
chain_spec_command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}polkadot build-spec --chain rococo-local --disable-default-bootnode"

[relaychain.default_resources]
limits = { memory = "4G", cpu = "2" }
Expand Down Expand Up @@ -45,89 +46,89 @@ requests = { memory = "2G", cpu = "1" }
[[parachains]]
id = 2000
addToGenesis = true
genesis_state_generator = "undying-collator export-genesis-state --pov-size=100000 --pvf-complexity=1"
genesis_state_generator = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator export-genesis-state --pov-size=100000 --pvf-complexity=1"

[parachains.collator]
name = "collator01"
image = "{{COL_IMAGE}}"
command = "undying-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator"
args = ["-lparachain=debug", "--pov-size=100000", "--pvf-complexity=1", "--parachain-id=2000"]

[[parachains]]
id = 2001
addToGenesis = true
genesis_state_generator = "undying-collator export-genesis-state --pov-size=100000 --pvf-complexity=10"
genesis_state_generator = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator export-genesis-state --pov-size=100000 --pvf-complexity=10"

[parachains.collator]
name = "collator02"
image = "{{COL_IMAGE}}"
command = "undying-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator"
args = ["-lparachain=debug", "--pov-size=100000", "--parachain-id=2001", "--pvf-complexity=10"]

[[parachains]]
id = 2002
addToGenesis = true
genesis_state_generator = "undying-collator export-genesis-state --pov-size=100000 --pvf-complexity=100"
genesis_state_generator = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator export-genesis-state --pov-size=100000 --pvf-complexity=100"

[parachains.collator]
name = "collator03"
image = "{{COL_IMAGE}}"
command = "undying-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator"
args = ["-lparachain=debug", "--pov-size=100000", "--parachain-id=2002", "--pvf-complexity=100"]

[[parachains]]
id = 2003
addToGenesis = true
genesis_state_generator = "undying-collator export-genesis-state --pov-size=20000 --pvf-complexity=300"
genesis_state_generator = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator export-genesis-state --pov-size=20000 --pvf-complexity=300"

[parachains.collator]
name = "collator04"
image = "{{COL_IMAGE}}"
command = "undying-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator"
args = ["-lparachain=debug", "--pov-size=20000", "--parachain-id=2003", "--pvf-complexity=300"]

[[parachains]]
id = 2004
addToGenesis = true
genesis_state_generator = "undying-collator export-genesis-state --pov-size=100000 --pvf-complexity=300"
genesis_state_generator = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator export-genesis-state --pov-size=100000 --pvf-complexity=300"

[parachains.collator]
name = "collator05"
image = "{{COL_IMAGE}}"
command = "undying-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator"
args = ["-lparachain=debug", "--pov-size=100000", "--parachain-id=2004", "--pvf-complexity=300"]

[[parachains]]
id = 2005
addToGenesis = true
genesis_state_generator = "undying-collator export-genesis-state --pov-size=20000 --pvf-complexity=400"
genesis_state_generator = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator export-genesis-state --pov-size=20000 --pvf-complexity=400"

[parachains.collator]
name = "collator06"
image = "{{COL_IMAGE}}"
command = "undying-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator"
args = ["-lparachain=debug", "--pov-size=20000", "--pvf-complexity=400", "--parachain-id=2005"]

[[parachains]]
id = 2006
addToGenesis = true
genesis_state_generator = "undying-collator export-genesis-state --pov-size=100000 --pvf-complexity=300"
genesis_state_generator = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator export-genesis-state --pov-size=100000 --pvf-complexity=300"

[parachains.collator]
name = "collator07"
image = "{{COL_IMAGE}}"
command = "undying-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator"
args = ["-lparachain=debug", "--pov-size=100000", "--pvf-complexity=300", "--parachain-id=2006"]

[[parachains]]
id = 2007
addToGenesis = true
genesis_state_generator = "undying-collator export-genesis-state --pov-size=100000 --pvf-complexity=300"
genesis_state_generator = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator export-genesis-state --pov-size=100000 --pvf-complexity=300"

[parachains.collator]
name = "collator08"
image = "{{COL_IMAGE}}"
command = "undying-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator"
args = ["-lparachain=debug", "--pov-size=100000", "--pvf-complexity=300", "--parachain-id=2007"]

[types.Header]
Expand Down
6 changes: 3 additions & 3 deletions zombienet_tests/functional/0001-parachains-pvf.zndsl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ eve: reports histogram polkadot_pvf_preparation_time has at least 1 samples in b
one: reports histogram polkadot_pvf_preparation_time has at least 1 samples in buckets ["0.1", "0.5", "1", "2", "3", "10"] within 10 seconds
two: reports histogram polkadot_pvf_preparation_time has at least 1 samples in buckets ["0.1", "0.5", "1", "2", "3", "10"] within 10 seconds

# Check all buckets >= 20.
# Check all buckets >= 20.
alice: reports histogram polkadot_pvf_preparation_time has 0 samples in buckets ["20", "30", "60", "180", "+Inf"] within 10 seconds
bob: reports histogram polkadot_pvf_preparation_time has 0 samples in buckets ["20", "30", "60", "180", "+Inf"] within 10 seconds
charlie: reports histogram polkadot_pvf_preparation_time has 0 samples in buckets ["20", "30", "60", "180", "+Inf"] within 10 seconds
Expand All @@ -78,8 +78,8 @@ two: reports histogram polkadot_pvf_preparation_time has 0 samples in buckets ["
# APPROVAL_EXECUTION_TIMEOUT(6s). Currently these are not differentiated by metrics
# because the metrics are defined in `polkadot-node-core-pvf` which is a level below
# the relevant subsystems.
# That being said, we will take the simplifying assumption of testing only the
# 2s timeout.
# That being said, we will take the simplifying assumption of testing only the
# 2s timeout.
# We do this check by ensuring all executions fall into bucket le="2" or lower.
# First, check if we have at least 1 sample, but we should have many more.
alice: reports histogram polkadot_pvf_execution_time has at least 1 samples in buckets ["0.1", "0.5", "1", "2"] within 10 seconds
Expand Down
12 changes: 6 additions & 6 deletions zombienet_tests/functional/0002-parachains-disputes.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ timeout = 1000
[relaychain]
default_image = "{{ZOMBIENET_INTEGRATION_TEST_IMAGE}}"
chain = "rococo-local"
chain_spec_command = "polkadot build-spec --chain rococo-local --disable-default-bootnode"
default_command = "polkadot"
chain_spec_command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}polkadot build-spec --chain rococo-local --disable-default-bootnode"
default_command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}polkadot"

[relaychain.default_resources]
limits = { memory = "4G", cpu = "2" }
Expand All @@ -18,13 +18,13 @@ requests = { memory = "2G", cpu = "1" }
[[relaychain.nodes]]
image = "{{MALUS_IMAGE}}"
name = "alice"
command = "malus dispute-ancestor --fake-validation approval-invalid"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}malus dispute-ancestor --fake-validation approval-invalid"
args = [ "--alice", " -lparachain=debug,MALUS=trace" ]

[[relaychain.nodes]]
image = "{{MALUS_IMAGE}}"
name = "bob"
command = "malus dispute-ancestor --fake-validation approval-invalid"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}malus dispute-ancestor --fake-validation approval-invalid"
args = [ "--bob", "-lparachain=debug,MALUS=trace"]

[[relaychain.nodes]]
Expand Down Expand Up @@ -55,12 +55,12 @@ requests = { memory = "2G", cpu = "1" }
[[parachains]]
id = {{id}}
addToGenesis = true
genesis_state_generator = "undying-collator export-genesis-state --pov-size={{25000*(id-1999)}} --pvf-complexity={{id - 1999}}"
genesis_state_generator = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator export-genesis-state --pov-size={{25000*(id-1999)}} --pvf-complexity={{id - 1999}}"

[parachains.collator]
image = "{{COL_IMAGE}}"
name = "collator"
command = "undying-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator"
args = ["-lparachain=debug", "--pov-size={{25000*(id-1999)}}", "--parachain-id={{id}}", "--pvf-complexity={{id - 1999}}"]

{% endfor %}
Expand Down
2 changes: 1 addition & 1 deletion zombienet_tests/functional/0003-beefy-and-mmr.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ timeout = 1000
[relaychain]
default_image = "{{ZOMBIENET_INTEGRATION_TEST_IMAGE}}"
chain = "rococo-local"
command = "polkadot"
default_command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}polkadot"

[[relaychain.node_groups]]
name = "validator"
Expand Down
12 changes: 6 additions & 6 deletions zombienet_tests/functional/0003-parachains-garbage-candidate.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ bootnode = true
[relaychain]
default_image = "{{ZOMBIENET_INTEGRATION_TEST_IMAGE}}"
chain = "rococo-local"
chain_spec_command = "polkadot build-spec --chain rococo-local --disable-default-bootnode"
default_command = "polkadot"
chain_spec_command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}polkadot build-spec --chain rococo-local --disable-default-bootnode"
default_command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}polkadot"

[relaychain.default_resources]
limits = { memory = "4G", cpu = "2" }
Expand All @@ -24,19 +24,19 @@ requests = { memory = "2G", cpu = "1" }
[[relaychain.node_groups]]
image = "{{MALUS_IMAGE}}"
name = "malus-validator"
command = "malus suggest-garbage-candidate"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}malus suggest-garbage-candidate"
args = ["-lparachain=debug,MALUS=trace"]
count = 1

{% for id in range(2000,2003) %}
[[parachains]]
id = {{id}}
addToGenesis = true
genesis_state_generator = "undying-collator export-genesis-state --pov-size={{10000*(id-1999)}} --pvf-complexity={{id - 1999}}"
genesis_state_generator = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator export-genesis-state --pov-size={{10000*(id-1999)}} --pvf-complexity={{id - 1999}}"
[parachains.collator]
image = "{{COL_IMAGE}}"
name = "collator"
command = "undying-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator"
args = ["-lparachain=debug", "--pov-size={{10000*(id-1999)}}", "--parachain-id={{id}}", "--pvf-complexity={{id - 1999}}"]
{% endfor %}

Expand Down
10 changes: 5 additions & 5 deletions zombienet_tests/misc/0001-paritydb.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ bootnode = true
[relaychain]
default_image = "{{ZOMBIENET_INTEGRATION_TEST_IMAGE}}"
chain = "rococo-local"
chain_spec_command = "polkadot build-spec --chain rococo-local"
default_command = "polkadot"
chain_spec_command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}polkadot build-spec --chain rococo-local"
default_command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}polkadot"

[relaychain.default_resources]
limits = { memory = "4G", cpu = "2" }
Expand All @@ -20,16 +20,16 @@ requests = { memory = "2G", cpu = "1" }
name = "validator"
count = 10
args = ["-lparachain=debug", "--db=paritydb"]

{% for id in range(2000,2010) %}
[[parachains]]
id = {{id}}
addToGenesis = true
genesis_state_generator = "undying-collator export-genesis-state --pov-size={{10000*(id-1999)}} --pvf-complexity={{id - 1999}}"
genesis_state_generator = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator export-genesis-state --pov-size={{10000*(id-1999)}} --pvf-complexity={{id - 1999}}"
[parachains.collator]
name = "collator"
image = "{{COL_IMAGE}}"
command = "undying-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator"
args = ["-lparachain=debug", "--pov-size={{10000*(id-1999)}}", "--parachain-id={{id}}", "--pvf-complexity={{id - 1999}}"]

{% endfor %}
Expand Down
5 changes: 3 additions & 2 deletions zombienet_tests/misc/0002-upgrade-node.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ timeout = 1000

[relaychain]
default_image = "{{ZOMBIENET_INTEGRATION_TEST_IMAGE}}"
default_command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}polkadot"
chain = "rococo-local"


Expand Down Expand Up @@ -30,7 +31,7 @@ addToGenesis = true
[parachains.collator]
name = "collator01"
image = "{{COL_IMAGE}}"
command = "undying-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator"
args = ["-lparachain=debug"]

[[parachains]]
Expand All @@ -40,7 +41,7 @@ addToGenesis = true
[parachains.collator]
name = "collator02"
image = "{{COL_IMAGE}}"
command = "undying-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}undying-collator"
args = ["-lparachain=debug"]

[types.Header]
Expand Down
4 changes: 2 additions & 2 deletions zombienet_tests/smoke/0001-parachains-smoke-test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ timeout = 1000
[relaychain]
default_image = "{{ZOMBIENET_INTEGRATION_TEST_IMAGE}}"
chain = "rococo-local"
command = "polkadot"
default_command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}polkadot"

[[relaychain.nodes]]
name = "alice"
Expand All @@ -21,7 +21,7 @@ addToGenesis = false
[parachains.collator]
name = "collator01"
image = "{{COL_IMAGE}}"
command = "/usr/local/bin/adder-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}adder-collator"
args = [ "-lruntime=debug,parachain=trace" ]

[types.Header]
Expand Down
4 changes: 2 additions & 2 deletions zombienet_tests/smoke/0002-parachains-upgrade-smoke-test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ bootnode = true
[relaychain]
default_image = "{{ZOMBIENET_INTEGRATION_TEST_IMAGE}}"
chain = "rococo-local"
command = "polkadot"
default_command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}polkadot"

[[relaychain.nodes]]
name = "alice"
Expand All @@ -31,7 +31,7 @@ cumulus_based = true
[parachains.collator]
name = "collator01"
image = "{{COL_IMAGE}}"
command = "polkadot-collator"
command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}polkadot-collator" # Should become `polkadot-parachain`

[[parachains.collator.env]]
name = "RUST_LOG"
Expand Down
3 changes: 1 addition & 2 deletions zombienet_tests/smoke/0003-deregister-register-validator-smoke.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ timeout = 1000
[relaychain]
default_image = "{{ZOMBIENET_INTEGRATION_TEST_IMAGE}}"
chain = "rococo-local"
command = "polkadot"
default_command = "{{ZOMBIENET_NATIVE_BINARY_PREFIX}}polkadot"

[[relaychain.nodes]]
name = "alice"
Expand All @@ -21,4 +21,3 @@ command = "polkadot"
[[relaychain.nodes]]
name = "dave"
args = [ "-lruntime=debug,parachain=trace" ]