Skip to content

Commit bf94de7

Browse files
authored
fix(l2): make batch-commit resends able to replace, and de-flake the shared-bridge test (#7151)
**Motivation** `Integration Test Shared Bridge - Normal` fails intermittently with `Receiver balance did not increase correctly` at `test/tests/l2/shared_bridge.rs:409`. It hit `main` on Aug 3 (`0cc4572c5`) and Aug 13 (`f2998963d`), and `release/v24.0.0` on Aug 12 (`8f3d2ca05`), always with the same message hash `0x66867250…5be951` — so the test is not wrong about what it expects, the message genuinely never arrives. In the failing runs L2B never commits batch 1, its L2→L2 message never becomes pending on the router, and L2A's watcher logs `L2 message not found in pending messages` 153 times with 0 verified. Three separate defects combine to produce that: **1. Both sequencers commit from the same L1 account.** `crates/l2/docker-compose.yaml` gives `ethrex_l2` `--committer.l1-private-key 0x385c…` and the shared-bridge override gave `ethrex_l2_b` the same key (likewise the proof-coordinator key). They post to one L1 on the same 15 s timer, so their commit loops race for the same nonce and the loser is rejected — first as an underpriced replacement, then as a too-low nonce. **2. The loser cannot recover, which is a bug beyond CI.** `bump_gas_generic_tx` computed the blob fee as `1 + (bump_percentage / 100) * 10` and then divided by ten. For the only bump either call site passes, `30`, that evaluates to `fee * 1 / 10`: **every retry lowered `max_fee_per_blob_gas` by 90%**, so a batch commitment could never re-price itself — on CI or on a real L1 during a fee spike. Blob replacements also require the fee to *double* (`DEFAULT_BLOB_PRICE_BUMP_PERCENT`), and that threshold applies to the fee cap and the tip as well, so bumping those by 30% could not have replaced a blob transaction either. Separately, the retry loop never noticed when the nonce had become unreachable, so one `NonceTooLow` burned all ten attempts in ~3.5 s and reported "max retries reached", hiding the cause. **3. The test slept a fixed 180 s and then asserted**, which made the result depend on whatever else the job was doing. The first failure in this job's history is the run of #7029, which aligned the `cargo test --no-run` prebuild's features with the run step so the test binary is no longer rebuilt inside it. Job step timestamps show the window between "Start Sequencer L2B" and the assertion collapsing from **2 m 35 s to 3 s**, moving the assertion onto the sequencers' cold start — where batch 1 carries all 192 privileged deposit mints and the two co-nonced committers collide from their first commit. Worth noting the direction: the passing Jul 22 run had *more* nonce collisions (72 underpriced + 76 too-low) than any failing run, because they happened before the test looked. **Description** - **L2B gets its own committer and proof-coordinator accounts, and its deployment authorizes them.** Changing the keys alone is not enough: `Timelock.commitBatch` is `onlyRole(SEQUENCER)`, and the role is granted from the deployer's `--committer.l1-address` / `--proof-sender.l1-address`, which both default to L2A's accounts. Without the matching addresses every L2B `commitBatch` reverts at gas estimation as an unauthorized caller — the same end symptom, reached a different way. Neither option is forwarded to the deployer container by `docker-compose.yaml`, so they are set on the override's `contract_deployer`, leaving L2A's deployment on the defaults. - **`bump_gas_generic_tx` raises every fee dimension** by a bump that clears the mempool's replacement threshold for the transaction's category — `BLOB_FEE_BUMP_PERCENT` (100) for blob-carrying transactions, `FEE_BUMP_PERCENT` (30) otherwise — using saturating arithmetic so an absurd starting fee cannot overflow. - **The retry loop stops spending attempts that cannot succeed:** it abandons the transaction with `UnreachableNonce` once the chain's nonce has passed it (checked against the chain, not against the rejection text, which differs between clients), and reports the reason when the `maximum_allowed_max_fee_per_gas` / `_blob_gas` clamps leave a bid the mempool already rejected. - **Every shared-bridge assertion now waits for its own condition** instead of sharing one fixed sleep, with a timeout more generous than the 180 s it replaces. The L1 deposits accounting is waited on separately rather than assumed settled once the L2B credit lands: the two are updated by different transactions on different chains and nothing orders them. The forced-inclusion test's expiry and verification waits are deliberately left as sleeps — they are waiting for something *not* to happen. - **Hardening:** a test pins L2B's private keys to the addresses its deployment authorizes, so drift between the two blocks of the override file fails there instead of surfacing at runtime as an ABI-undecodable `AccessControlUnauthorizedAccount` revert and a chain that silently stops committing. Unit tests cover both replacement thresholds, monotonicity across ten retries, saturation, and — so they cannot silently stop discriminating — that the previous blob formula fails them. This is deliberately **not** a revert of #7029. That PR removed an accidental delay which had been masking defects 1 and 2 since the shared-bridge test landed; restoring the delay would hide them again. **Also worth knowing, not changed here:** this job is gated `github.event_name != 'merge_group'`, so the merge queue never runs it and a PR can land without it ever executing against the merged result. Combined with the paths filter, it effectively only runs on pushes to main — which is why a ~3% flake sat unnoticed for weeks. **Checklist** - [x] Updated `STORE_SCHEMA_VERSION` (crates/storage/lib.rs) if the PR includes breaking changes to the `Store` requiring a re-sync. — n/a, no `Store` changes
1 parent ac8eb46 commit bf94de7

7 files changed

Lines changed: 506 additions & 40 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎crates/l2/docker-compose-l2-shared-bridge.overrides.yaml‎

Lines changed: 24 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,15 @@ services:
1010
- /bin/bash
1111
- -c
1212
- touch /env/.env; ./ethrex l2 deploy "$0" "$@"
13+
# The Timelock is initialized with the committer and proof-sender addresses
14+
# as its authorized sequencers, and both deployer options default to the
15+
# accounts ethrex_l2 uses. L2B runs from its own accounts (see ethrex_l2_b
16+
# below), so its deployment has to authorize those instead, or every
17+
# commitBatch from L2B reverts at gas estimation as an unauthorized caller.
18+
# These two addresses must stay in sync with ethrex_l2_b's private keys.
19+
environment:
20+
- ETHREX_DEPLOYER_COMMITTER_L1_ADDRESS=0x000990B05481b1661bc6211298f6429451B09425
21+
- ETHREX_DEPLOYER_PROOF_SENDER_L1_ADDRESS=0x000cDF8Dba2393a40857cbCB0FCD9b998a941078
1322
command: >
1423
--use-compiled-genesis false
1524
--genesis-l2-path ${DOCKER_ETHREX_WORKDIR}/fixtures/genesis/l2.json
@@ -62,15 +71,28 @@ services:
6271
- -c
6372
- export $(xargs < /env/.env); ./ethrex l2 "$0" "$@"
6473
# ETHREX_WATCHER_BRIDGE_ADDRESS and ETHREX_COMMITTER_ON_CHAIN_PROPOSER_ADDRESS are set in the .env file by the contract_deployer service.
74+
#
75+
# The L1 keys below MUST differ from the ones ethrex_l2 uses in
76+
# docker-compose.yaml. Both sequencers post to the same L1, so two
77+
# committers on one account race for the same nonce, and the loser cannot
78+
# recover: its send is rejected as an underpriced replacement, then as a
79+
# too-low nonce until it exhausts its retries, and it re-wedges on the same
80+
# batch every cycle. One chain then never commits, so its L2->L2 messages
81+
# never become pending on the router and the shared-bridge test observes no
82+
# balance change on the destination chain.
83+
#
84+
# These are rich accounts from fixtures/keys/private_keys_l1.txt. Their only
85+
# other L1 use is the deployer's rich deposit, which is sent from each rich
86+
# account itself but completes before either sequencer starts.
6587
command: >
6688
--network /genesis/l2.json
6789
--http.addr 0.0.0.0
6890
--http.port 1730
6991
--authrpc.port 8553
7092
--proof-coordinator.addr 0.0.0.0
7193
--block-producer.coinbase-address 0x0007a881CD95B1484fca47615B64803dad620C8d
72-
--committer.l1-private-key 0x385c546456b6a603a1cfcaa9ec9494ba4832da08dd6bcf4de9a71e4a01b74924
73-
--proof-coordinator.l1-private-key 0x39725efee3fb28614de3bacaffe4cc4bd8c436257e2c8bb887c4b5c4be45e76d
94+
--committer.l1-private-key 0xfc8ae6457da1052f379f9a7eb925e11d10d3fc1dc0b59fe93658704babcfe78c
95+
--proof-coordinator.l1-private-key 0xfdb73112480f2b033ff2fe2e6f5bd358fc46307a3fc2aeb70c79e77a8704ad82
7496
--datadir dev_ethrex_l2b
7597
--proof-coordinator.port 3901
7698
--no-monitor

‎crates/l2/sdk/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ repository.workspace = true
1010

1111
[dependencies]
1212
ethrex-common.workspace = true
13+
ethrex-blockchain.workspace = true
1314
ethrex-rpc.workspace = true
1415
ethrex-l2-common.workspace = true
1516
ethrex-l2-rpc.workspace = true

0 commit comments

Comments
 (0)