Turn list of txs into tx sequence #683

aaronbuchwald · 2023-06-17T19:43:50Z

Why this should be merged

This PR abstracts the load simulator in a different way.

The previous version used a series of workers that performed the tasks of 1) issuing transactions, 2) waiting for the last accepted nonce to exceed the last nonce of the latest transaction, and 3) confirming the transaction receipts in order.

It then created N workers in order to issue and confirm all of the transactions to separate RPC endpoints. This new version switches to using a TxSequence that returns a channel of transactions that need to be issued and confirmed and a worker that plays the role of issuing and confirming transactions.

On top of these two, we can create different issuance strategies to play around with. In this PR, those two different strategies are:

issue all transactions at once and confirm them sequentially
Issue N transactions, confirm the whole batch, and repeat until iterating through the entire transaction sequence

There are two motivating factors for this change

Make it easier to simulate different issuance strategies
Simulate more realistic workloads where each tx agent makes a choice based off of the results of the transactions as opposed to issuing all transactions as a fire and forget operation, which is not a realistic use case.

An example of a more realistic pattern is for warp messaging. For the warp messaging PR #586 we would like to perform some degree of stress testing for sending/receiving warp messages.

The goal of this refactor is to make it as easy as possible to construct a realistic load simulation for warp that can perform the following process and scale up to stress the system:

Issue a transaction to send a warp message
Aggregate a warp signature and create a transaction to deliver the warp message

A fun version of this load test may be to create a very simple smart contract to send a monotonically increasing message back and forth between two different subnets.

How this was tested

This PR was tested locally with a variety of parameters including:

ws and rpc endpoints
multiple endpoints
single validator and multi-validator

How is this documented

The existing README for the simulator still documents how to use the load simulator correctly.

aaronbuchwald · 2023-06-17T20:58:40Z

cmd/simulator/load/workers.go

-	endTime := time.Now()
-	log.Info("Transaction confirmation completed", "confirmationElapsed", endTime.Sub(confirmationStart))
-	totalTxs := 0
-	for _, worker := range wg.workers {
-		totalTxs += len(worker.txs)
-	}
-
-	// We exclude the time for ConfirmAllTransactions because we assume once the nonce is reported, the transaction has been finalized.
-	// For Subnet-EVM, this assumes that unfinalized data is not exposed via API.
-	issueAndConfirmDuration := confirmationStart.Sub(executionStart)
-	log.Info("Simulator completed", "totalTxs", totalTxs, "TPS", float64(totalTxs)/issueAndConfirmDuration.Seconds())


TODO:

add back these same metrics

add metrics for individual transaction latency

in the next pr

ceyonur · 2023-06-19T09:02:47Z

cmd/simulator/txs/agent.go

+	n        int
+}
+
+func NewIssueNAgent[T any](sequence TxSequence[T], worker Worker[T], n int) Agent[T] {


do we use this?

ceyonur · 2023-06-19T09:03:40Z

cmd/simulator/txs/agent.go

+			tx   T
+			done bool
+		)
+		for i := 0; i < a.n; i++ {


How does this work for NewIssueAllAgent where n=0?

ceyonur · 2023-06-19T09:14:14Z

cmd/simulator/load/tx_generator.go

Should we move this to txs package?

ceyonur · 2023-06-19T09:14:50Z

cmd/simulator/load/tx_generator.go


 // GenerateTxSequence fetches the current nonce of key and calls [generator] [numTxs] times sequentially to generate a sequence of transactions.
-func GenerateTxSequence(ctx context.Context, generator CreateTx, client ethclient.Client, key *ecdsa.PrivateKey, numTxs uint64) ([]*types.Transaction, error) {
+func GenerateTxSequence(ctx context.Context, generator CreateTx, client ethclient.Client, key *ecdsa.PrivateKey, numTxs uint64) (txs.TxSequence[*types.Transaction], error) {


Do we need this to be exist/exported? I think GenerateTxSequences is just a generalized version of this.

i can unexport this

never mind it is used in funder

ceyonur · 2023-06-19T09:17:15Z

cmd/simulator/load/loader.go

 	signer := types.LatestSignerForChainID(chainID)

+	log.Info("Creating transaction sequences...")
 	txGenerator := func(key *ecdsa.PrivateKey, nonce uint64) (*types.Transaction, error) {


It seems this txGenerator func can just be a series of options/params for GenerateTxSequences

ceyonur · 2023-06-19T09:22:22Z

cmd/simulator/load/loader.go

 		}
 		return tx, nil
 	}
 	txSequences, err := GenerateTxSequences(ctx, txGenerator, clients[0], pks, config.TxsPerWorker)


Do you think Agent can handle generating tx sequences itself by taking priv keys + numTx?

cmd/simulator/load/loader.go

cmd/simulator/txs/agent.go

anusha-ctrl · 2023-06-20T16:05:50Z

cmd/simulator/load/worker.go

-		} else {
-			log.Info("fetched nonce", "awaiting", nonce, "currentNonce", currentNonce)
+			return fmt.Errorf("failed to await tx %s nonce %d: %w", tx.Hash(), txNonce, err)
 		}


can we still keep the awaiting line. It's helpful to the user.

I think it's excessive to do so for every nonce at the info level. Could we make it debug?

cmd/simulator/load/worker.go

anusha-ctrl · 2023-06-20T16:19:45Z

cmd/simulator/txs/agent.go

+	txChan := a.sequence.Chan()
+
+	txs := make([]T, 0, 100)
+	for tx := range txChan {


seems like there are two functions that do the same thing. One has a fixed N at 100 and one has a nonfixed N. We should keep the nonfixed N function and have the user decide N in config cli flags

darioush · 2023-06-20T17:13:28Z

cmd/simulator/load/worker.go

+	txNonce := tx.Nonce()
+
 	for {
+		// If the is less than what has already been accepted, the transaction is confirmed


Suggested change

// If the is less than what has already been accepted, the transaction is confirmed

// If the nonce is less than what has already been accepted, the transaction is confirmed

darioush · 2023-06-27T17:41:40Z

cmd/simulator/README.md

 1. Ignore stake weight on the P-Chain and count each connected peer as having a stake weight of 1
 2. Automatically opts in to validate every Subnet
-:::
+   :::


what does this change do?

my editor added that

aaronbuchwald · 2023-06-27T22:03:08Z

LGTM once the ::: is removed

* Turn list of txs into tx sequence * Add agent comments * Remove unused code and address comments * add some more comments * address more comments' * lint * Remove :::warning --------- Co-authored-by: Anusha <anushachillara@gmail.com>

* Rename test_ to step_ for clarity (ava-labs#636) * Codeowners fix (specific dir overrides global) (ava-labs#639) * rename state upgrade test (ava-labs#641) * plugin/evm: fix godoc typo (ava-labs#647) Signed-off-by: Gyuho Lee <gyuho.lee@avalabs.org> * plugin/evm: remove unncessary tx.Hash op in map iter (ava-labs#648) Signed-off-by: Gyuho Lee <gyuho.lee@avalabs.org> * deployExampleRewardManager.ts hardhat script fixed (ava-labs#650) * Migrate HardHat tests to DS-tests (Solidity tests) (ava-labs#601) * Increase trie dirty default cache size to 512MB (ava-labs#656) * export gingkgo suites (ava-labs#659) * export gingkgo suites * remove precompiles description from ping tests * update package.json * fix url in package json * change package name * Publish npm module for contract-examples (ava-labs#663) * export gingkgo suites * remove precompiles description from ping tests * update package.json * fix url in package json * change package name * prepare for module publishing * use npm ci * preload ethers * nits * Fix broken path to docs in README. (ava-labs#665) * Fix contract examples package json (ava-labs#672) * rename contract-examples to contracts & fix pack issue * allow custom CMD to be run on blockchains for e2e tests * move default hardhat test to utils * fix comment * add more comments * change folder reference * Bump got and ethereum-waffle in /contracts (ava-labs#680) Removes [got](https://github.com/sindresorhus/got). It's no longer used after updating ancestor dependency [ethereum-waffle](https://github.com/EthWorks/Waffle). These dependencies need to be updated together. Removes `got` Updates `ethereum-waffle` from 3.4.4 to 4.0.10 - [Release notes](https://github.com/EthWorks/Waffle/releases) - [Commits](https://github.com/EthWorks/Waffle/compare/ethereum-waffle@3.4.4...ethereum-waffle@4.0.10) --- updated-dependencies: - dependency-name: got dependency-type: indirect - dependency-name: ethereum-waffle dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump flat and hardhat in /contracts (ava-labs#679) Bumps [flat](https://github.com/hughsk/flat) to 5.0.2 and updates ancestor dependency [hardhat](https://github.com/nomiclabs/hardhat). These dependencies need to be updated together. Updates `flat` from 4.1.1 to 5.0.2 - [Commits](hughsk/flat@4.1.1...5.0.2) Updates `hardhat` from 2.6.1 to 2.15.0 - [Release notes](https://github.com/nomiclabs/hardhat/releases) - [Commits](https://github.com/nomiclabs/hardhat/compare/hardhat-core-v2.6.1...hardhat@2.15.0) --- updated-dependencies: - dependency-name: flat dependency-type: indirect - dependency-name: hardhat dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix core benchmarks (ava-labs#697) * remove unused line in test (ava-labs#695) * bump version to v0.5.2 (ava-labs#681) * bump version to v0.5.2 * downgrade avalanchego to 1.10.2 * update compatibility * Remove unused ethereum-waffle and ethereumjs-tx pkgs (ava-labs#685) * move state upgrade related functions to own file (ava-labs#684) * Update load test to use ANR for network setup (ava-labs#579) * Update load test to use ANR for network setup * Remove ToWebsocketURI that relies on strings feature added in go1.20 * Update load test to use ENV var from CI if populated * Move env var logic into NewDefaultANRConfig * Add init functions and clean up comments * Remove unnecessary helper * Remove unneeded cmd * Cleanup * Address PR comments * Move 2 min timeout for SetupNetwork to the start of the function * Turn list of txs into tx sequence (ava-labs#683) * Turn list of txs into tx sequence * Add agent comments * Remove unused code and address comments * add some more comments * address more comments' * lint * Remove :::warning --------- Co-authored-by: Anusha <anushachillara@gmail.com> * Minor simulator fixes (ava-labs#703) * add benchmark to template (ava-labs#705) * add benchmark to template * fix comment * Add metrics for gas price / usage (ava-labs#700) Co-authored-by: aaronbuchwald <aaron.buchwald56@gmail.com> * added config setting to prune warpdb (ava-labs#702) * added config setting to prune warpdb * refactored warp prune uint test * message * fixed up nits * fixed nits * fixed nits * Update plugin/evm/config.go * Add wrapped validator state for primary to subnet warp messages (ava-labs#708) * Add wrapped validator state for primary to subnet warp message verification * Address PR comment * upload network runner artifact after e2e test run (ava-labs#709) * Differentiate mandatory/optional Subnet-EVM Network Upgrades (ava-labs#640) * remove network upgrades from upgrade configs and enforce them in vm init * fix tests * remove unused var * remove extra shutdown * copy test chain config when modify * use SimulatedTestChainConfig for simulated tests * copy modified test chain config * add optional upgrades * Add tests * fix comments * use unmarshal in test * unexport functions * add commented out test * add TODO comment * add blockchain.Stop as defer to test (ava-labs#714) Co-authered-by: najeal <haim.nathan@icloud.com> Co-authored-by: Ceyhun Onur <ceyhun.onur@avalabs.org> * Add support for predicate gas into IntrinsicGas (ava-labs#713) * Add support for predicate gas * Address review * Fix test and add test for PredicateGas * Address comments * Add signature aggregation helpers (ava-labs#711) * Add signature aggregation helpers * Fix * Move WarpQuorumDenominator into new file * Address comments * fix var rename * Improve comments * Address comments * Add predicate packing helper (ava-labs#710) * Add predicate packing helper * move predicate code and add readme * fix moved import * fix merge * Run ginkgo precompile tests in parallel (ava-labs#712) Use SynchronizedBeforeSuite to create the node and all the subnet that are needed to run tests. Pass the blockchainID to each work processor and run the tests in parallel to reduce runtime * Fix BlockContext comment (ava-labs#731) * precompile accepter should take logIdx (ava-labs#728) * precompile accepter should take logIdx * minimize mock * copyright yr * add comments * Improve tests (ava-labs#733) Improve tests based on feedback from ava-labs#712 * Fix precompile test gen template (ava-labs#737) * fix Benchmark * fix spelling/references in Benchmark * Clean up gossip (ava-labs#744) * Remove call from precompile (ava-labs#748) * Fix nil types tmpl (ava-labs#751) * add more nil types and fix int types * readd hash type * remove convertToNil * Sync coreth v0.12.4 rc0 (ava-labs#694) * apply diff to rc0 * replace coreth with subnet-evm * go mod tidy * remove must commit * remove apricots * replace coreth with subnet-evm * rename banff and cortina * test memory db * add dUpgrade * fix errors * sync with v0.12.4-rc0 * more diffs * change templates * readd preparePredicateSlots * run goimports * readd tx allow list check to txpool * Fix tests * readd subnet evm check for tests * fix tests * add gas limit & coinbase fakers * Fix timestamp types * check error from newblockchain * fix gas limit in simulated genesis * downgrade ci to go 1.19 * gofmt files * rebump go to 1.20 * bump anr to latest * bump anr to latest * prepare predicate storage slots in Subnet-EVM * uncomment tests * change order of airdrop memory cleanup * add utils import * Coreth 0.12.4 x apply (ava-labs#720) Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> * move instruction sets * revert unpack changes * reduce diffs * move genesis verify to vm (ava-labs#753) * move genesis verify to vm * fix comment * set a correct default from genesis * fix log * revert change to vm.go --------- Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> * avoid adding/removing GasLimits in tests (ava-labs#755) * move genesis verify to vm * fix comment * set a correct default from genesis * fix log * revert change to vm.go * avoid adding/removing GasLimits in tests * nit --------- Co-authored-by: Ceyhun Onur <ceyhun.onur@avalabs.org> --------- Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> * Bump avalanchego to v1.10.5 and bump Subnet-EVM for v0.5.3 release (ava-labs#757) * Bump avalanchego to v1.10.5 and bump Subnet-EVM for v0.5.3 release * bump anr version to v1.7.1 * Migrate ABI unpack fix (ava-labs#750) * change sprintf verb and use values (ava-labs#760) * Generalize precompile test suite (ava-labs#759) * restore old functions, generalize paths in sync subnet creation * add subnet suit struct and methods * move constants * fix reviews * add comment * Update subnet.go Signed-off-by: Ceyhun Onur <ceyhun.onur@avalabs.org> --------- Signed-off-by: Ceyhun Onur <ceyhun.onur@avalabs.org> * Add warp contract implementation (ava-labs#718) * Add warp contract implementation * Cleanup predicate test * Fix new function signature * Replace invalid fuzz test with unit test * Add chain config to enable warp API for warp e2e test * remove unused var * Add experimental warning and move warp precompile to x/ package * fix warning label * Fix warning * vm test nits * Improve sendWarpMessenger sol comment * more vm warp test nits * Move warp params into params package * More vm warp test nits * Address more PR comments * Remove triggerTx2 * Add check for expected topics from sendWarpMessage log * Fix config test * Fix incorrect replace * remove unnecessary echo * Address comments * Address comments * Address PR comments * Improve comments * Convert [32]byte type to common.Hash * Add base cost for getVerifiedWarpMessage * fix require equal type check * Fix updated awm message format * Update warp message format * Move IWarpMessenger.sol to interfaces/ * Add newline to warp genesis * uncomment assertion * Fix broken links in README * Add metrics to simulator (ava-labs#706) * add tps * get block build metrics * Add issuance time, confirmed time, issued-> acceptance time, verify time * Log times at end of batch * cleaner * address comments * remove unused code block * avoids taking len of channel * nits * pass in stringID * move to loader * remove unused field * revert file back * cleaner * lint * make it work for ws or for rpc * protect * endpoint * no return on defer * sep to a funciton * have blockchainidstr passed in * typo * pass in metrics through config * address comments * address more comments and edit err policy of metrics functions * add more logging to load_test * typo * better check * fix endpoints * typo: * individual * histogram * address feedback: * remove metrics from default * address comments * simplify time metrics * better explanation * address comments * address comments * cleanup * more cleanup * rename vars for clarity * ws * cleanup * address comments * ws * expose metrics add flag * fix blocking issue of http server and gracefully stop it * cleanup * use constant * add issuance to confirmation metrics * ws * simplify metrics server * Bump avalanchego to v1.10.5 and bump Subnet-EVM for v0.5.3 release (ava-labs#757) * Bump avalanchego to v1.10.5 and bump Subnet-EVM for v0.5.3 release * bump anr version to v1.7.1 * handle control c * print out output * clean up * clean up * remove go routines to close client * address comments * memory leak * fix * print * e2e test: connect to appropriate chain (ava-labs#771) * Bump avalanchego dep and update version for next release (ava-labs#770) * Bump avalanchego dep and update version for next release * Update cache sizes * Update flag in run script to replace staking-enabled=false * Update block builder to enforce predicate one tx at a time (ava-labs#773) * Update block builder to enforce predicate one tx at a time * remove dead code * Update miner/worker.go Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> Signed-off-by: aaronbuchwald <aaron.buchwald56@gmail.com> --------- Signed-off-by: aaronbuchwald <aaron.buchwald56@gmail.com> Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> * update iocorders address * remove state-sync-enabled from chain config * 1s block ticker * flush hubbleFeed when len > 512 * fix compilation err --------- Signed-off-by: Gyuho Lee <gyuho.lee@avalabs.org> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Ceyhun Onur <ceyhun.onur@avalabs.org> Signed-off-by: aaronbuchwald <aaron.buchwald56@gmail.com> Co-authored-by: Richard Pringle <rpring9@gmail.com> Co-authored-by: Darioush Jalali <darioush.jalali@avalabs.org> Co-authored-by: Ceyhun Onur <ceyhun.onur@avalabs.org> Co-authored-by: Gyuho Lee <gyuho.lee@avalabs.org> Co-authored-by: İzzet Emre Demir <izzetemredemir@gmail.com> Co-authored-by: aaronbuchwald <aaron.buchwald56@gmail.com> Co-authored-by: Gabriel Cardona <gabriel@avalabs.org> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Anusha <anushachillara@gmail.com> Co-authored-by: Cesar <137245636+nytzuga@users.noreply.github.com> Co-authored-by: morrisettjohn <60852062+morrisettjohn@users.noreply.github.com> Co-authored-by: nathan haim <nathan.haim@free.fr> Co-authored-by: Martin Eckardt <m.eckardt@outlook.com> Co-authored-by: rodrigo <77309055+RodrigoVillar@users.noreply.github.com> Co-authored-by: Anusha <63559942+anusha-ctrl@users.noreply.github.com>

aaronbuchwald added 2 commits June 17, 2023 15:31

Turn list of txs into tx sequence

9a0b4fe

Add agent comments

024b54b

aaronbuchwald marked this pull request as ready for review June 17, 2023 20:25

aaronbuchwald requested review from ceyonur and darioush as code owners June 17, 2023 20:25

aaronbuchwald commented Jun 17, 2023

View reviewed changes

ceyonur reviewed Jun 19, 2023

View reviewed changes

anusha-ctrl reviewed Jun 20, 2023

View reviewed changes

cmd/simulator/load/loader.go Show resolved Hide resolved

anusha-ctrl reviewed Jun 20, 2023

View reviewed changes

cmd/simulator/txs/agent.go Show resolved Hide resolved

anusha-ctrl reviewed Jun 20, 2023

View reviewed changes

cmd/simulator/load/worker.go Show resolved Hide resolved

anusha-ctrl reviewed Jun 20, 2023

View reviewed changes

anusha-ctrl added 4 commits June 27, 2023 09:50

Remove unused code and address comments

774320f

add some more comments

61ff4b2

address more comments'

fabe777

lint

ae385ba

anusha-ctrl requested a review from ceyonur June 27, 2023 18:56

darioush reviewed Jun 27, 2023

View reviewed changes

darioush approved these changes Jun 27, 2023

View reviewed changes

aaronbuchwald added 2 commits June 27, 2023 18:04

Remove :::warning

4639809

Merge branch 'master' into simulator-sequence-txs

c3192ed

aaronbuchwald merged commit c2f010b into master Jun 27, 2023

aaronbuchwald deleted the simulator-sequence-txs branch June 27, 2023 22:06

darioush pushed a commit that referenced this pull request Dec 3, 2024

libevm renaming nits (#683)

bfcc2f6

	// If the is less than what has already been accepted, the transaction is confirmed
	// If the nonce is less than what has already been accepted, the transaction is confirmed

Turn list of txs into tx sequence #683

Turn list of txs into tx sequence #683

Uh oh!

Conversation

aaronbuchwald commented Jun 17, 2023

Why this should be merged

How this was tested

How is this documented

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aaronbuchwald commented Jun 27, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants