Refactor Workload #1668

samliok · 2024-10-16T23:07:43Z

I approached this refactor by considering how a VM developer would want to test their VM. This PR is not the final refactor of the testing framework for the HyperSDK, but I believe it's a good starting point to discuss what that framework should be.

Changes

Reduces the interface VM developers need to implement to just one method that generates transactions. This is a significant simplification compared to the previous TxWorkloadFactory, which required implementing two functions that, in turn, required the user to implement TxWorkloadIterators with an additional two functions.

type TxAssertion func(ctx context.Context, require *require.Assertions, uri string)
type TxGenerator interface {
    GenerateTx(context.Context, string) (*chain.Transaction, TxAssertion, error)
}

Generating "workloads" is now handled under the hood by the TxWorkload struct that has one TxGenerator field.
This simplification means all a developer needs to do is implement the TxGenerator interface and pass it to the e2e tests to sanity check their VM with the HyperSDK's e2e suite(well that and include a run script, cmd, and other extra stuff that should be abstracted soon).

Reasoning & Design

Current e2e testing served as both HyperSDK tests and tests for MorpheusVM, which added layers of complexity and cognitive load for devs.
We already have e2e tests that are testing both the HyperSDK and MorpheusVM. Let's make these e2e tests generic and easily digestible for developers.
We want to offer a variety of testing experiences to developers on the HyperSDK. Procedural testing(switch from workload generator to procedural test style for VM-defined workloads #1359) for VMs should be included in addition to this form of testing.

examples/morpheusvm/tests/generator/mixed.go

x/contracts/runtime/util_test.go

tsachiherman

Maybe I'm missing some file, but I can't find any use of NewMultiAuthTxGenerator or multiAuthTxGenerator.

examples/morpheusvm/cmd/morpheus-cli/cmd/chain.go

joshua-kim · 2024-10-22T14:37:43Z

tests/fixture/consts.go

+)
+
+// hardcoded initial set of ed25519 keys. Each will be initialized with InitialBalance
+var ed25519HexKeys = []string{


Why do we hard-code these instead of generating them from newDefaultKeys?

tests/integration/integration.go

joshua-kim · 2024-10-22T14:59:00Z

tests/workload/transactions.go

+type TxGenerator interface {
+	// GenerateTx generates a new transaction and an assertion function that confirms
+	// the transaction was accepted by the network
+	GenerateTx(context.Context, string) (*chain.Transaction, TxAssertion, error)


Another idea is if the action developer should even have to generate an assertion for each tx.

If we want to check for the tx against the indexer api I think we can actually define that entirely in the calling code since we know the indexer url + the tx id instead of the action dev defining this.

We could however expect the action dev to generate an assertion for the final resulting state keys that should exist after all txs are applied though.

We also probably want to think about how this framework should be able to support multiple txs per block and ordering in execution but that's probably out of scope for this PR.

examples/morpheusvm/tests/generator/mixed.go

joshua-kim · 2024-10-22T15:32:30Z

examples/morpheusvm/tests/generator/simple.go

+	_, tx, _, err := cli.GenerateTransaction(
+		ctx,
+		parser,
+		[]chain.Action{&actions.Transfer{
+			To:    aother,
+			Value: 1,
+		}},
+		g.factory,
+	)


I wonder if using the apis to determine our max fees here is overkill and if we could just have the caller specifying a constant max fee it's willing to pay. Could be considered out of scope for this PR though.

tests/fixture/keys.go

tests/fixture/vm.go

aaronbuchwald

Nice cleanup on the tx generator interface 👍

I don't think we should add NewTestVM this way because it groups together the genesis definition and ed25519 auth package as if they will be used by every VM, which is not the case (VMs can select which auth types to use or add their own and do not need to include ed25519).

Could we separate moving the shared genesis definition into tests/fixture/ from this PR?

tests/e2e/e2e.go

tests/workload/transactions.go

tests/fixture/vm.go

aaronbuchwald

LGTM - we can add back the functionality of Workloads(...) via #1667

tests/workload/transactions.go

aaronbuchwald · 2024-10-23T14:54:57Z

tests/workload/transactions.go


-	for generator.Next() && height < targetheight {
-		tx, confirm, err := generator.GenerateTxWithAssertion(ctx)
+	for count := 0; count < blocks && height < targetheight; count++ {


Why are we using count this way? This function should just ensure that at least blocks were generated, it doesn't need to ensure that we generated and confirmed that number of transactions.

I could extend this function to take in another argument txPerBlock, this is the same as the previous logic for generating blocks. Is it worth doing this is a follow up?

This was actually a looser condition before, not more exact by requiring a specific number of transactions. We can produce blocks with no transactions in them and this function is intended to guarantee at least that number of blocks were produced, but only checks the height to allow empty blocks to be included.

Could we remove the extra check and leave this as only height < targetHeight ?

Co-authored-by: aaronbuchwald <aaron.buchwald56@gmail.com> Signed-off-by: Sam Liokumovich <65994425+samliok@users.noreply.github.com>

merge approved.

tsachiherman · 2024-10-22T15:57:39Z

tests/fixture/vm.go

+	"github.com/ava-labs/hypersdk/genesis"
+)
+
+type testVM struct {


the content of testVM doesn't really looks anything like a VM. I think that it should be changed, but wouldn't mind making this change as part of my subsequent PR.

commit b2ad4d3 Author: Tsachi Herman <24438559+tsachiherman@users.noreply.github.com> Date: Sun Nov 3 16:01:37 2024 -0500 support procedural test style for VM-defined workloads (#1667) Signed-off-by: Tsachi Herman <24438559+tsachiherman@users.noreply.github.com> commit dbcc6c9 Author: Richard Pringle <richard.pringle@avalabs.org> Date: Thu Oct 31 16:42:44 2024 -0400 Update wasmtime-go (#1705) commit 1015ae1 Author: Sam Liokumovich <65994425+samliok@users.noreply.github.com> Date: Thu Oct 31 12:47:45 2024 -0400 Remove HRP + Decimals dead code (#1703) commit 24bacf8 Author: Sam Liokumovich <65994425+samliok@users.noreply.github.com> Date: Wed Oct 30 10:31:38 2024 -0400 Action Test Reusing Store (#1700) commit 1ce6ad8 Author: Richard Pringle <richard.pringle@avalabs.org> Date: Tue Oct 29 13:23:56 2024 -0400 Don't benchmark serializing parameters (#1697) commit 1c43f7e Author: Sam Liokumovich <65994425+samliok@users.noreply.github.com> Date: Tue Oct 29 13:16:34 2024 -0400 Call Contract Benchmarks (#1696) commit 7bd1a49 Author: nathan haim <haim.nathan@icloud.com> Date: Sat Oct 26 20:41:26 2024 +0200 state: Make state.Keys JSON human readable (#1661) Signed-off-by: nathan haim <nathan.haim@free.fr> commit c592a1d Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com> Date: Fri Oct 25 14:02:08 2024 -0400 Add `balance` cmd to CLI (#1691) commit 7fac7fa Author: Sam Liokumovich <65994425+samliok@users.noreply.github.com> Date: Fri Oct 25 13:28:52 2024 -0400 Default Contract Manager Implementation (#1687) commit 0459bd4 Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com> Date: Fri Oct 25 12:23:04 2024 -0400 Add Getter to `BalanceHandler` (#1685) commit 960b790 Author: containerman17 <8990432+containerman17@users.noreply.github.com> Date: Sat Oct 26 00:14:29 2024 +0900 Universal CLI (#1662) Creates a single binary that can be used with any HyperSDK VM, as long as the VM uses standard functionality. Signed-off-by: containerman17 <8990432+containerman17@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Aaron Buchwald <aaron.buchwald56@gmail.com> commit ef66271 Author: Richard Pringle <richard.pringle@avalabs.org> Date: Thu Oct 24 15:45:17 2024 -0400 Move vmwithcontracts out of examples (#1676) commit 3895ff5 Author: Richard Pringle <richard.pringle@avalabs.org> Date: Thu Oct 24 14:59:57 2024 -0400 Implement NFT example (#1684) commit 82dc621 Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com> Date: Thu Oct 24 09:54:31 2024 -0400 Add back `actionID` to `StateKeys()` (#1683) commit cf6d48a Author: Richard Pringle <richard.pringle@avalabs.org> Date: Wed Oct 23 15:11:25 2024 -0400 Force test-context to be initialized with an actor (#1682) commit 1a3c670 Author: Sam Liokumovich <65994425+samliok@users.noreply.github.com> Date: Wed Oct 23 14:26:09 2024 -0400 Refactor Workload (#1668) Signed-off-by: Sam Liokumovich <65994425+samliok@users.noreply.github.com> Co-authored-by: aaronbuchwald <aaron.buchwald56@gmail.com> commit 403bfb3 Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com> Date: Tue Oct 22 21:56:25 2024 -0400 Decouple State Manager (#1658) commit 61993f8 Author: Richard Pringle <richard.pringle@avalabs.org> Date: Mon Oct 21 14:25:25 2024 -0400 Revert "Make Morpheus build script take optional path (#1671)" (#1673) commit 87cf88a Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Mon Oct 21 12:39:13 2024 -0400 Update min go version to go1.22.8 (#1672) commit 22241f5 Author: Richard Pringle <richard.pringle@avalabs.org> Date: Fri Oct 18 14:18:53 2024 -0400 Make Morpheus build script take optional path (#1671) commit 0658976 Author: Richard Pringle <richard.pringle@avalabs.org> Date: Wed Oct 16 18:37:50 2024 -0400 Move runtime tests into contracts-ci (#1666) commit e3d911a Author: Sam Liokumovich <65994425+samliok@users.noreply.github.com> Date: Sun Oct 13 17:45:32 2024 -0400 Remove global variables from spam script (#1659) commit bca7739 Author: Tsachi Herman <24438559+tsachiherman@users.noreply.github.com> Date: Fri Oct 11 11:41:27 2024 -0400 Create SignedTransaction out of the existing Transaction (#1640) commit 6f15b2f Author: Sam Liokumovich <65994425+samliok@users.noreply.github.com> Date: Thu Oct 10 20:47:49 2024 -0400 Include Spam Config Defaults (#1657) commit b3713a1 Author: Sam Liokumovich <65994425+samliok@users.noreply.github.com> Date: Thu Oct 10 20:16:23 2024 -0400 Separate Spam Broadcast Functionality (#1653) commit 182e023 Author: Sam Liokumovich <65994425+samliok@users.noreply.github.com> Date: Thu Oct 10 18:19:44 2024 -0400 Decouple Spam Functionality from CLI into Throughput Package (#1636) commit b8104ac Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Thu Oct 10 14:44:08 2024 -0400 Fix incorrect naming of stateless block (#1652) commit 725a589 Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Thu Oct 10 10:12:59 2024 -0400 Use CreateActionID to give simulated actions unique actionID (#1644) commit 0a2d991 Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Thu Oct 10 09:57:25 2024 -0400 Update ActionRegistry to ActionCodec (#1650) commit 829b991 Author: containerman17 <8990432+containerman17@users.noreply.github.com> Date: Thu Oct 10 22:42:19 2024 +0900 Reflect Marshaler (#1592) Signed-off-by: containerman17 <8990432+containerman17@users.noreply.github.com> commit b3c991f Author: Tsachi Herman <24438559+tsachiherman@users.noreply.github.com> Date: Thu Oct 10 09:36:21 2024 -0400 add simulateTransaction endpoint (#1610) Signed-off-by: Tsachi Herman <24438559+tsachiherman@users.noreply.github.com> commit cbb6a33 Author: containerman17 <8990432+containerman17@users.noreply.github.com> Date: Thu Oct 10 22:24:22 2024 +0900 Add boolean support to the ABI spec (#1648) Signed-off-by: containerman17 <8990432+containerman17@users.noreply.github.com> commit 5d8c590 Author: Sam Liokumovich <65994425+samliok@users.noreply.github.com> Date: Thu Oct 10 09:21:46 2024 -0400 Remove marshaling from transfer action (#1647) commit b934dbd Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Date: Wed Oct 9 16:52:29 2024 -0400 Remove type alias for Action + Auth + Output Registry (#1547) Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> commit b5407c9 Author: containerman17 <8990432+containerman17@users.noreply.github.com> Date: Thu Oct 10 00:45:38 2024 +0900 Simulate a chain of actions (#1635) commit fee360f Author: Tsachi Herman <24438559+tsachiherman@users.noreply.github.com> Date: Fri Oct 4 11:23:58 2024 -0400 refactor marshalActions implementation (#1631) commit 2cb5530 Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Thu Oct 3 18:49:12 2024 -0400 Add block indexing to Indexer API (#1606) commit f1bcc59 Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Thu Oct 3 18:31:39 2024 -0400 Add JSON marshalling to Result (#1627) commit 2ea784a Author: Richard Pringle <richard.pringle@avalabs.org> Date: Thu Oct 3 12:54:10 2024 -0400 Implement multisig example (#1581) commit 6bbf236 Author: Richard Pringle <richard.pringle@avalabs.org> Date: Thu Oct 3 12:43:41 2024 -0400 Update codeowners (#1629) commit 47849d8 Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Thu Oct 3 08:34:51 2024 -0400 Re-mark pubsub tests as flaky (#1625) commit d847132 Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Wed Oct 2 14:23:24 2024 -0400 Reduce tests.unit.sh timeout value less than CI job (#1623) commit 404e74f Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Wed Oct 2 14:07:04 2024 -0400 Add JSON marshalling for fee dimensions (#1622) commit 7e53003 Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Date: Wed Oct 2 12:38:34 2024 -0400 Replace usage of `internal/network` with `p2p` (#1613) Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> commit 79f363e Author: containerman17 <8990432+containerman17@users.noreply.github.com> Date: Wed Oct 2 11:53:59 2024 +0900 Standardize decimals (#1620) commit e191c88 Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Date: Tue Oct 1 20:02:30 2024 -0400 Update to avalanchego@v1.11.12-rc.2 (#1614) Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> commit 59c84d7 Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Tue Oct 1 17:19:17 2024 -0400 Add ExecutedBlock type (#1601) commit 165a455 Author: Tsachi Herman <24438559+tsachiherman@users.noreply.github.com> Date: Tue Oct 1 09:26:32 2024 -0400 validate buffer length prior to calling Uint64 (#1588) commit c24f0d8 Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com> Date: Tue Oct 1 09:25:18 2024 -0400 Fix VM-With-Contracts Unit Tests (#1602) commit 8ab9d83 Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Mon Sep 30 16:57:29 2024 -0400 Minor nits on pubsub implementation (#1593) commit 592cd7a Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com> Date: Mon Sep 30 12:46:35 2024 -0400 Remove `StateKeysMaxChunks()` (#1607) commit 3e358c1 Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Mon Sep 30 12:46:20 2024 -0400 Update ActionBenchmark to use single ExpectedOutput (#1608) commit e78841e Author: Franfran <51274081+iFrostizz@users.noreply.github.com> Date: Mon Sep 30 16:42:00 2024 +0200 Write FIFO cache unit tests (#1201) * write FIFO cache unit tests * invert expected / actual * add limit parameter and cache fail on 0 size * simplify test table and create new test for empty cache * simpler tests by having a limit of 2 * fix test cases * typos * backout uselesss change * add a "not an LRU" test case * use switch in fifo unit tests commit f1150a1 Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Fri Sep 27 17:46:28 2024 -0400 Update tx indexer to include tx action outputs (#1597) commit 6a3fd63 Author: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> Date: Thu Sep 26 15:44:56 2024 -0400 Remove usage of mockgen (#1591) Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com> commit 49376bd Author: rodrigo <77309055+RodrigoVillar@users.noreply.github.com> Date: Thu Sep 26 14:39:59 2024 -0400 Fix benchmarks (#1590) commit 77a73ba Author: Richard Pringle <richard.pringle@avalabs.org> Date: Tue Sep 24 12:38:27 2024 -0400 Ignore gas on mock-function-call (#1585) commit 1bbab7b Author: containerman17 <8990432+containerman17@users.noreply.github.com> Date: Tue Sep 24 23:19:18 2024 +0900 Remove special cases for Bytes, add arrays support (#1587) commit 3410599 Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Mon Sep 23 12:34:55 2024 -0400 Add spam cmd to morpheus readme (#1577) commit dfefd27 Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Mon Sep 23 12:34:32 2024 -0400 Improve chain comments (#1579) commit 6c15244 Author: Richard Pringle <richard.pringle@avalabs.org> Date: Fri Sep 20 16:10:21 2024 -0400 Series of changes that improve development (#1583) commit e0a3324 Author: Richard Pringle <richard.pringle@avalabs.org> Date: Fri Sep 20 13:12:58 2024 -0400 Use repr(packed) with state-keys (#1580) commit 75c6cfd Author: aaronbuchwald <aaron.buchwald56@gmail.com> Date: Thu Sep 19 07:58:39 2024 -0400 Separate proposer monitor from vm into internal package (#1574) Signed-off-by: Joshua Kim <20001595+joshua-kim@users.noreply.github.com>

samliok added 28 commits October 8, 2024 14:24

updated vm with contracts actions + storage + vm

72d4f3a

remove assets + mod tidy

c5da65f

deploy statekeys

66ea4e9

merge main

1c0dffa

progress on tests, saving

94930b7

code clarifications

68e158c

Merge branch 'main' into updated-contracts-vm

f1631cd

remove unmarshaling

1f19257

e2e tests ported

347ab65

Merge branch 'main' into updated-contracts-vm

1cb76f6

tx sending but errduplicatetx

861b2d9

remove integration

29030c5

simplify workload

15e1987

move into tests package

fded8d0

move to fixtures

7069ead

rename initial balance

c0d9783

initial balance

dded573

Merge branch 'boilerplate' into updated-contracts-vm

234f9e2

merge

c0fa333

decent devx for creating a tx, now need to run it

ea0683b

potentially clean up workload

bba6987

refactor workloads

8d4bd14

i think new refactor is working

338baad

reduce generator interface area

b9d39ee

reduce even more

26b0058

remove globals

dceb881

deploy transaction working

84083a7

workload changes

d084fbc

samliok self-assigned this Oct 16, 2024

samliok requested a review from richardpringle as a code owner October 16, 2024 23:07

reset to simple

5f06bff

samliok commented Oct 17, 2024

View reviewed changes

examples/morpheusvm/tests/generator/mixed.go Outdated Show resolved Hide resolved

samliok commented Oct 18, 2024

View reviewed changes

x/contracts/runtime/util_test.go Show resolved Hide resolved

samliok and others added 5 commits October 18, 2024 15:19

lint

8dad4fe

Merge branch 'main' into workload-refactor

4ee7ef6

reduce diff

6b881c7

lint mixed auth generator

a8add01

lint

3c74ccf

tsachiherman previously requested changes Oct 22, 2024

View reviewed changes

examples/morpheusvm/cmd/morpheus-cli/cmd/chain.go Outdated Show resolved Hide resolved

joshua-kim reviewed Oct 22, 2024

View reviewed changes

samliok and others added 5 commits October 22, 2024 13:04

remove Ed25519 key type

efaf3a4

const interval

61706ae

readd unit tests

8859af6

perms

d20c932

Merge branch 'main' into workload-refactor

803ae57

aaronbuchwald requested changes Oct 22, 2024

View reviewed changes

tests/e2e/e2e.go Outdated Show resolved Hide resolved

tests/workload/transactions.go Outdated Show resolved Hide resolved

tests/workload/transactions.go Outdated Show resolved Hide resolved

tests/fixture/vm.go Outdated Show resolved Hide resolved

samliok and others added 4 commits October 22, 2024 15:14

create workload in morpheus

dfc0c01

remove vm file

c8ab82b

decouple key type

b80c24e

Merge branch 'main' into workload-refactor

2bae438

aaronbuchwald previously approved these changes Oct 23, 2024

View reviewed changes

Update tests/workload/transactions.go

ce9b199

Co-authored-by: aaronbuchwald <aaron.buchwald56@gmail.com> Signed-off-by: Sam Liokumovich <65994425+samliok@users.noreply.github.com>

samliok dismissed aaronbuchwald’s stale review via ce9b199 October 23, 2024 15:13

remove count

eab0bb2

aaronbuchwald approved these changes Oct 23, 2024

View reviewed changes

tsachiherman approved these changes Oct 23, 2024

View reviewed changes

samliok merged commit 1a3c670 into main Oct 23, 2024
18 checks passed

Refactor Workload #1668

Refactor Workload #1668

Uh oh!

Conversation

samliok commented Oct 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Reasoning & Design

Uh oh!

Uh oh!

Uh oh!

tsachiherman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joshua-kim Oct 22, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joshua-kim Oct 22, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joshua-kim Oct 22, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aaronbuchwald left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aaronbuchwald left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aaronbuchwald Oct 23, 2024

Choose a reason for hiding this comment

Uh oh!

samliok Oct 23, 2024

Choose a reason for hiding this comment

Uh oh!

aaronbuchwald Oct 23, 2024

Choose a reason for hiding this comment

Uh oh!

tsachiherman Oct 22, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

samliok commented Oct 16, 2024 •

edited

Loading