Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Roadmap v2 #410

Closed
wants to merge 26 commits into from
+39 −0
Closed

Roadmap v2 #410

Changes from 20 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
3c0616e
add roadmap
pepoviola Sep 20, 2022
801f476
more items
pepoviola Sep 20, 2022
575cbf5
Update roadmap.md
wirednkod Sep 20, 2022
f6e94fc
Update roadmap.md
wirednkod Sep 20, 2022
143ae1f
Update roadmap.md
wirednkod Sep 20, 2022
e9c47e9
Update roadmap.md
wirednkod Sep 20, 2022
a74bf78
Update roadmap.md
wirednkod Sep 20, 2022
69384c6
Update roadmap.md
pepoviola Sep 20, 2022
3c3a90e
Update roadmap.md
pepoviola Sep 20, 2022
5df62b3
Update roadmap.md
pepoviola Sep 20, 2022
b39fc6f
Update roadmap.md
pepoviola Sep 20, 2022
32b9690
Update roadmap.md
pepoviola Sep 21, 2022
12b98e0
add item
pepoviola Sep 21, 2022
9a63297
Update roadmap.md
pepoviola Sep 22, 2022
e85b5fc
Update roadmap.md
wirednkod Sep 23, 2022
ff29047
Update roadmap.md
wirednkod Sep 29, 2022
94857cc
Merge branch 'main' into roadmap-v2
pepoviola Sep 29, 2022
a81f8e4
Update roadmap.md
pepoviola Sep 30, 2022
2a948a1
Update roadmap.md
wirednkod Sep 30, 2022
e35b697
Update roadmap.md
wirednkod Sep 30, 2022
529237c
Merge branch 'main' into roadmap-v2
pepoviola Oct 24, 2022
bc37954
Update roadmap.md
pepoviola Oct 29, 2022
2d5c211
Update roadmap.md
pepoviola Oct 29, 2022
fedbe08
Merge branch 'main' into roadmap-v2
wirednkod Nov 24, 2022
45cc22f
Update roadmap.md
wirednkod Feb 27, 2023
683ea1e
Merge branch 'main' into roadmap-v2
wirednkod Apr 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions roadmap.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
Roadmap Zombienet v2

## Infra
Copy link
Contributor

@sandreim sandreim Sep 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prometheus server deployment -> asserting against prometheus queries.

  • we remove code and inefficiency from scraping metrics ourselves
  • scales better
  • we get a standardized way of querying metrics and ability to do aggregations as we see fit
  • (nice to have) create some test reports using these, maybe alarms for long duration failure conditions
  • local debugability increases

Open question:

  • do we use prometheus for native/local runs, or just k8s ? (if this differers across providers, prometheus queries won't work)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think local prometheus should be opt-in and not a default.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Long lived test networks deployment and management.

  • deploy and manage Versi using Zombienet - similar functionality with validator manager
  • obsoletes our custom solution currently built and maintained by devops

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deployment scalability improvements - up to 1000 validators.

- Chaos testing, add examples and explore possibilities in `native` and `podman` provider
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is top priority for parachains. We want to roll out a separate CI pipeline to do these long duration tests.

- Add `docker` provider
- Add `nomad` provider
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @drahnr, did you have time to make a quick chat to review some request relates to nomad?
Thanks!

- Create [helm chart](https://helm.sh/docs/topics/charts/) to allow other use zombienet in k8s
- Auth system to not use k8s users
- Create GitHub Action and publish in marketplace (wip)
pepoviola marked this conversation as resolved.
Show resolved Hide resolved
- Rename `@paritytech/zombienet` npm package to `zombienet`. Keep all zombienet modules under `@zombienet/*` org
wirednkod marked this conversation as resolved.
Show resolved Hide resolved

## Internal teams
- Add more teams (wip)

## UI
- Create UI to create `.zndls` and `network` files.
- Improve VSCode extension (grammar/snippets/syntax highlighting/file validations) ([repo](https://github.com/paritytech/zombienet-vscode-extension))
- Create UI app (desktop) to run zombienet without the need of terminal.

## Registry
- Create decorators registry and allow override by paras (wip)
- Explore how to get info from paras.

## Functional tasks
Copy link
Contributor

@sandreim sandreim Sep 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Zombienet Test SDK (Rust) - it's not replacing the deployment which remains as is (typescript) + additional features listed above. Integration point is the output JSON file from deployment which is used to spawn the Zombienet Test SDK test environment.

Advantages of this approach:

  • one language to fully write the entire test (deployment part + actual tests to be performed)
  • simple APIs to generate the Network file - builder pattern
  • simple APIs to query/assert metrics
  • write more complex test logic
  • directly use or wrap subxt - open HRMP channels, send XCM, do runtime upgrades, etc
  • opens up the Rust ecosystem of libraries that can be used in a test
  • feature parity with current DSL support (phasing out DSL over time after we have same featureset)
  • easier for people outside Parity to contribute to the project

- Add subxt integration, allow to compile/run on the fly
- Move parser to pest (wip)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️ happy to collab on this :)

Copy link
Collaborator Author

@pepoviola pepoviola Sep 21, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @drahnr, I started the work in this draft pr. The parser is already working and I need to finish the wiring between the parser crate and the test-runner. If you have time to review the pr would be awesome :)

Thanks!

- Detach phases and use JSON to communicate instead of `paths`
- Add relative values assertions (for metrics/scripts)
- Allow to define nodes that are not started in the launching phase and can be started by the test-runner
- Allow to define `race` assertions
- Rust integration -> Create multiples libs (crates)
- Explore backchannel use case
- Add support to run test agains a running network (wip)
- Add more CLI subcommands
- Add js/subxt snippets ready to use in assertions (e.g transfers)
- Add XCM support in built-in assertions
pepoviola marked this conversation as resolved.
Show resolved Hide resolved
- Add support to start from a live network (fork-off) [check subalfred]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From our experience (already doing fork-off Moonbeam chains):
This will require chain-specific actions (changing the authoring values, changing council, changing balance....). So a proper way to interact with the raw state is needed for each case.
It also requires to deal with very large database (or exported state json file) (Moonbeam > 5GB), which is not often easily done.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should perhaps file sub-issues to all roadmap items for further discussion once finalized.

Certainly, this is a desirable feature. The main motivation I see is to test node or runtime changes against live network states as a pre-release check. This does require some ability to interact with raw state. To start from a live network state, the two major things are to allow swapping out the runtime (optional) and to allow modification of the state.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should perhaps file sub-issues to all roadmap items for further discussion once finalized.

Certainly, this is a desirable feature. The main motivation I see is to test node or runtime changes against live network states as a pre-release check. This does require some ability to interact with raw state. To start from a live network state, the two major things are to allow swapping out the runtime (optional) and to allow modification of the state.

Thanks for your feedback @rphmeier! I will create a couple of umbrella issues to capture the vision/plan for the public roadmap and individual issues for items.

Thanks!

- Create "default configuration" - (if `zombieconfig.json` exists in same dir with zombienet then the config applied in it will override the default configuration of zombienet. E.G if user wants to have as default `native` instead of `k8s` he can add to default the entry `{ default: 'native' }`.)