devnet-9: 1000-node mainnet-distribution stress test

[barnabasbusa]

Summary

Devnet-9 stress-tests the devnet-8 setup at mainnet scale: 1000 nodes split by approximate mainnet client distribution, 1M active validators (1000/node), and 3M exited validators sprayed into genesis for a 4M-entry registry.

Terraform (terraform/devnet-9)

Copy of devnet-8's structure (network glamsterdam-devnet-9, own state key, CIDR 10.29.0.0/16) with a rewritten node matrix:

CL \ EL	geth 41%	nethermind 30%	reth 14%	besu 8%	erigon 4%	ethrex 2%	nimbusel 1%	Σ
lighthouse 43%	176	129	60	35	17	9	4	430
prysm 31%	127	93	44	25	12	6	3	310
teku 14%	57	42	20	11	6	2	2	140
nimbus 8%	33	24	11	6	3	2	1	80
lodestar 3%	13	9	4	2	1	1	0	30
grandine 1%	4	3	1	1	1	0	0	10

Largest-remainder rounding so rows and columns hit targets exactly; contiguous validator ranges in file order over [0,1000000), verified gap-free. Plus 4 bootnodes and the same 4 buildoor combos as devnet-8 (builder indices 0-3). CL weights from ethereum.org's client-diversity snapshot, EL from ethernodes-style node counts, with small ethrex/nimbusel slices for coverage.

Ansible (ansible/inventories/devnet-9)

Copied from devnet-8 minus files/ and inventory.ini (terraform generates it). Key changes:

• NUMBER_OF_VALIDATORS=1000000; exited-spray of 3M validators (status 2, BLS creds, inert) at mnemonic indices 1.1M+, block-shuffled into the active set via SHUFFLE_VALIDATORS (mapping lands in metadata/validator_names.yaml)
• Genesis pinned 2026-08-24 10:00 UTC; Gloas at epoch 16, 200M gas bump at epoch 32
• EIP-8282 contracts deployed in genesis (no window for the assertoor deploy playbook with Gloas at epoch 16), premines dropped
• Client images stay on the devnet-8 tags (stress test of the same builds)
• Config distributed via eth_testnet_config_tarball_* pull instead of controller push — genesis.ssz lands at ~560MB, so rsync x1000 hosts is a non-starter

Before launch

• Upload the metadata tarball after genesis generation (command in all.yaml)
• Regenerate bootnode keys in bootnode.sops.yaml (copied from devnet-8)
• DO droplet quota for 1009 instances (+ cost check: 1000x supernode size)
• genesis.ssz / parsed json must stay out of git (~560MB / ~1.3GB)

[redpandabot]

Summary

Config-only PR that copies the devnet-8 structure to a 1000-node / 1M-validator mainnet-scale stress test: node matrix and validator ranges verified exactly contiguous over [0,1M) with the claimed CL/EL distribution, genesis pinned with Gloas at epoch 16, 3M exited validators sprayed via egg 6.2.0's ADDITIONAL_VALIDATOR_MNEMONICS + SHUFFLE_VALIDATORS (all confirmed supported by the pinned egg version), and config distribution switched from controller rsync to a CDN tarball because genesis.ssz grows to ~560MB. No blockers found; two concerns worth a look before launch.

Issues

• 🟡 ansible/inventories/devnet-9/group_vars/all/all.yaml:102 — SHUFFLE_VALIDATORS orphans the per-host validator ranges behind the inventory-web / buildoor endpoints — This is the first devnet in the repo to enable SHUFFLE_VALIDATORS. eth_inventory_web_segments_json (line 124) still publishes per-host ranges [start,end) from ethereum_genesis_validator_keyranges, and those ranges are what /meta/api/v1/validator-ranges.json feeds to dora, assertoor, beacon-metrics-gazer and buildoor's --validator-ranges-url. Once the 3M exited spray is block-shuffled in, key-derivation indices no longer equal on-chain validator indices, so those consumers would display a host's validators at the wrong indices unless they resolve ranges through the validator_names.yaml mapping shipped in the tarball. I could not trace how the downstream consumers map ranges to on-chain indices — worth confirming the mapping is consulted before launch.
• 🟡 ansible/inventories/devnet-9/group_vars/all/all.yaml:100 — Nothing enforces the 'keep genesis.ssz out of git' rule, and the repo convention commits it — The comment says to keep the ~560MB genesis.ssz / ~1.3GB parsed JSON out of git, but .gitignore has no entry for them (unlike network-configs/**/mnemonics.yaml and **/files/validator_keys/*), and every prior devnet's genesis.ssz and parsedConsensusGenesis.json are tracked in this repo (verified via git ls-files: devnet-7 genesis.ssz 69MB, devnet-8 parsed json 39MB). A routine git add network-configs/devnet-9 after genesis generation would stage a >100MB blob and hard-fail the push under GitHub's cap. Consider adding a .gitignore entry before this lands.

---

_{Reviewed @ 5644dce4
"A child is a deadline with a face."}