Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Roadmap v2 #410

Closed
wants to merge 26 commits into from
Closed
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions roadmap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
Roadmap Zombienet v2

## Infra
Copy link
Contributor

@sandreim sandreim Sep 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prometheus server deployment -> asserting against prometheus queries.

  • we remove code and inefficiency from scraping metrics ourselves
  • scales better
  • we get a standardized way of querying metrics and ability to do aggregations as we see fit
  • (nice to have) create some test reports using these, maybe alarms for long duration failure conditions
  • local debugability increases

Open question:

  • do we use prometheus for native/local runs, or just k8s ? (if this differers across providers, prometheus queries won't work)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think local prometheus should be opt-in and not a default.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Long lived test networks deployment and management.

  • deploy and manage Versi using Zombienet - similar functionality with validator manager
  • obsoletes our custom solution currently built and maintained by devops

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deployment scalability improvements - up to 1000 validators.

- Chaos testing, add examples and explore possibilities in `native` and `podman` provider
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is top priority for parachains. We want to roll out a separate CI pipeline to do these long duration tests.

- Add `docker` provider
- Add `nomad` provider
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @drahnr, did you have time to make a quick chat to review some request relates to nomad?
Thanks!

- Create helm chart to allow other use zombienet in k8s
pepoviola marked this conversation as resolved.
Show resolved Hide resolved
- Auth system to not use k8s users
- Create GitHub Action and publish in marketplace
pepoviola marked this conversation as resolved.
Show resolved Hide resolved

## Intenal teams
wirednkod marked this conversation as resolved.
Show resolved Hide resolved
- Add more teams (wip)

## UI
- Create UI to create `feature` and `network` file.
pepoviola marked this conversation as resolved.
Show resolved Hide resolved
- Create script to translate from polkadot launch config to zombienet network file.
- Improve vscode extension (grammar/snippets)
wirednkod marked this conversation as resolved.
Show resolved Hide resolved

## Registry
- Create decorators registry and allow override by paras (wip)
- Explore how to get info from paras.

## Functional tasks
Copy link
Contributor

@sandreim sandreim Sep 20, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Zombienet Test SDK (Rust) - it's not replacing the deployment which remains as is (typescript) + additional features listed above. Integration point is the output JSON file from deployment which is used to spawn the Zombienet Test SDK test environment.

Advantages of this approach:

  • one language to fully write the entire test (deployment part + actual tests to be performed)
  • simple APIs to generate the Network file - builder pattern
  • simple APIs to query/assert metrics
  • write more complex test logic
  • directly use or wrap subxt - open HRMP channels, send XCM, do runtime upgrades, etc
  • opens up the Rust ecosystem of libraries that can be used in a test
  • feature parity with current DSL support (phasing out DSL over time after we have same featureset)
  • easier for people outside Parity to contribute to the project

- Add subxt integration, allow to compile/run on the fly
- Move parser to pest (wip)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️ happy to collab on this :)

Copy link
Collaborator Author

@pepoviola pepoviola Sep 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @drahnr, I started the work in this draft pr. The parser is already working and I need to finish the wiring between the parser crate and the test-runner. If you have time to review the pr would be awesome :)

Thanks!

- Detach phases and use JSON to communicate instead of `paths`
- Add relative values assertions (for metrics/scripts)
- Allow to define nodes that are not started in the launching phase and can be started by the test-runner
- Allow to define `race` assertions
- Rust integration -> Create multiples libs (crates)
- Explore backchannel use case
- Add support to run test agains a running network (wip)
- Add more CLI subcommands
- Add js/subxt snippets ready to use in assertions (e.g transfers)
- Add XCM support in built-in assertions
pepoviola marked this conversation as resolved.
Show resolved Hide resolved