Skip to content

fix(operator): manage subscription to events without panic #1729

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

JuArce
Copy link
Collaborator

@JuArce JuArce commented Jan 9, 2025

Manage Operator's event subscription

Description

Previously, the Operator panicked if one connection (either main or fallback) failed.

The previous solution to this (#1692) was to iterate infinitely over both connections, this had the "silent failure" problem, as an operator could fail and retry forever without notice to its owner.

Now if one connection fails, the Operator will use the other one, while continiously trying to reconnect to the failed one; and if both connections fail, and the retryables are consumed, the Operator will exit, logging the errors of each connection RPC, so the owner/manager of the Operator server can fix it accordingly.

This PR also added the usage of fallback when operator calls DisableVerifiers

To test

  1. Start Ethereum Package:
make ethereum_package_start
  1. Start verification layer components:
make aggregator_start_ethereum_package ENVIRONMENT=devnet
make batcher_start_ethereum_package
make operator_register_and_start_ethereum_package ENVIRONMENT=devnet
  1. Stop and Start the containers ethereum package started for reth-2 and reth-3. reth-3 is set as the fallback of the operator and reth-2 as the main node:
docker stop <NODE_ID>
docker start <NODE_ID>
  • Verify the Operator behaves as described, and while submitting proofs take down 1 connection, take down the other connection, take down both connections.

Type of change

Please delete options that are not relevant.

  • New feature
  • Bug fix
  • Optimization
  • Refactor

Checklist

  • “Hotfix” to testnet, everything else to staging
  • Linked to Github Issue
  • This change depends on code or research by an external entity
    • Acknowledgements were updated to give credit
  • Unit tests added
  • This change requires new documentation.
    • Documentation has been added/updated.
  • This change is an Optimization
    • Benchmarks added/run
  • Has a known issue
  • If your PR changes the Operator compatibility (Ex: Upgrade prover versions)
    • This PR adds compatibility for operator for both versions and do not change batcher/docs/examples
    • This PR updates batcher and docs/examples to the newer version. This requires the operator are already updated to be compatible

@uri-99 uri-99 marked this pull request as ready for review January 13, 2025 14:36
Copy link
Contributor

@uri-99 uri-99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

biased self-approve

@Oppen Oppen added audit cantina Audit report from Cantina labels Jan 20, 2025
@JuArce JuArce removed audit cantina Audit report from Cantina labels Jan 20, 2025
Copy link
Collaborator Author

@JuArce JuArce left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same comments apply for V3

@MarcosNicolau MarcosNicolau changed the base branch from testnet to staging May 20, 2025 17:42
Copy link

Changes to gas cost

Generated at commit: a33d8a04aa35a8606bcd45a8a35bc7dea911b4c8, compared to commit: b77e8fdf474834f0b9d018c28322b5c29512002e

🧾 Summary (10% most significant diffs)

Contract Method Avg (+/-) %
RegistryCoordinatorHarness blsApkRegistry
stakeRegistry
+354 ❌
+354 ❌
+57.47%
+51.91%
StakeRegistryHarness delegation +146 ❌ +22.19%

Full diff report 👇
Contract Deployment Cost (+/-) Method Min (+/-) % Avg (+/-) % Median (+/-) % Max (+/-) % # Calls (+/-)
Slasher 728,578 (-870,247) initialize 366 (-556) -60.30% 366 (-556) -60.30% 366 (-556) -60.30% 366 (-556) -60.30% 6 (0)
RegistryCoordinatorHarness 5,528,683 (-4,601,377) blsApkRegistry
initialize
stakeRegistry
970 (+354)
55,078,420 (-1,203,055)
1,036 (+354)
+57.47%
-2.14%
+51.91%
970 (+354)
55,078,420 (-1,203,055)
1,036 (+354)
+57.47%
-2.14%
+51.91%
970 (+354)
55,078,420 (-1,203,055)
1,036 (+354)
+57.47%
-2.14%
+51.91%
970 (+354)
55,078,420 (-1,203,055)
1,036 (+354)
+57.47%
-2.14%
+51.91%
6 (0)
6 (0)
6 (0)
StakeRegistryHarness 2,590,172 (-2,549,338) delegation
initializeQuorum
804 (+146)
144,420 (-1,780)
+22.19%
-1.22%
804 (+146)
164,216 (-1,780)
+22.19%
-1.07%
804 (+146)
164,320 (-1,780)
+22.19%
-1.07%
804 (+146)
164,320 (-1,780)
+22.19%
-1.07%
6 (0)
1,152 (0)
AlignedLayerServiceManager 4,398,072 (-3,921,136) batchesState
createNewTask
disableVerifier
disabledVerifiers
enableVerifier
isVerifierDisabled
receive
setDisabledVerifiers
5,132 (-278)
56,970 (-824)
24,602 (+303)
3,030 (+516)
23,748 (-443)
2,406 (-450)
23,301 (-195)
24,243 (-7)
-5.14%
-1.43%
+1.25%
+20.53%
-1.83%
-15.76%
-0.83%
-0.03%
5,132 (-278)
76,868 (-920)
36,191 (+375)
3,030 (+516)
24,388 (-372)
2,406 (-450)
46,835 (-356)
35,262 (+86)
-5.14%
-1.18%
+1.05%
+20.53%
-1.50%
-15.76%
-0.75%
+0.24%
5,132 (-278)
77,026 (-920)
36,191 (+375)
3,030 (+516)
24,388 (-372)
2,406 (-450)
47,115 (-357)
35,262 (+86)
-5.14%
-1.18%
+1.05%
+20.53%
-1.50%
-15.76%
-0.75%
+0.24%
5,132 (-278)
77,812 (-956)
47,780 (+446)
3,030 (+516)
25,029 (-300)
2,406 (-450)
47,115 (-357)
46,282 (+179)
-5.14%
-1.21%
+0.94%
+20.53%
-1.18%
-15.76%
-0.75%
+0.39%
256 (0)
256 (0)
2 (0)
1 (0)
2 (0)
3 (0)
256 (0)
2 (0)
AVSDirectory 1,485,692 (-1,689,090) initialize 97,570 (-3,067) -3.05% 97,570 (-3,067) -3.05% 97,570 (-3,067) -3.05% 97,570 (-3,067) -3.05% 6 (0)
ServiceManagerMock 1,340,500 (-1,292,283) initialize 71,296 (-2,066) -2.82% 71,296 (-2,066) -2.82% 71,296 (-2,066) -2.82% 71,296 (-2,066) -2.82% 6 (0)
ProxyAdmin 412,495 (-345,151) upgrade
upgradeAndCall
38,758 (-883)
55,403,611 (-1,228,407)
-2.23%
-2.17%
38,767 (-883)
55,403,611 (-1,228,407)
-2.23%
-2.17%
38,770 (-883)
55,403,611 (-1,228,407)
-2.23%
-2.17%
38,770 (-883)
55,403,611 (-1,228,407)
-2.23%
-2.17%
24 (0)
6 (0)
BLSApkRegistryHarness 1,676,721 (-1,447,516) initializeQuorum
setBLSPublicKey
45,382 (-596)
89,551 (-556)
-1.30%
-0.62%
45,382 (-596)
89,551 (-556)
-1.30%
-0.62%
45,382 (-596)
89,551 (-556)
-1.30%
-0.62%
45,382 (-596)
89,551 (-556)
-1.30%
-0.62%
1,152 (0)
6 (0)
TransparentUpgradeableProxy 521,501 (-376,498) fallback 1,634 (+90) +5.83% 47,449 (-617) -1.28% 8,200 (+90) +1.11% 119,180 (-820) -0.68% 30 (0)
StrategyManagerMock 1,146,070 (-1,255,302) setAddresses 88,974 (-768) -0.86% 88,974 (-768) -0.86% 88,974 (-768) -0.86% 88,974 (-768) -0.86% 6 (0)
IndexRegistry 917,389 (-790,168) initializeQuorum 45,244 (-348) -0.76% 45,244 (-348) -0.76% 45,244 (-348) -0.76% 45,244 (-348) -0.76% 1,152 (0)

@JuArce JuArce changed the title hotfix(operator): manage subscription to events without panic fix(operator): manage subscription to events without panic May 20, 2025
Copy link
Member

@MarcosNicolau MarcosNicolau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worked for me. I've updated the how-to-test section in the pr description to use ethereum-package. Not related to this pr I noticed that when deactivating the fallback of the Batcher, the Batcher stops listening to blocks, thus not creating new batches.

MarcosNicolau and others added 2 commits May 28, 2025 16:11
Co-authored-by: Marcos Nicolau <76252340+MarcosNicolau@users.noreply.github.com>
@MauroToscano MauroToscano added this pull request to the merge queue Jun 4, 2025
Merged via the queue into staging with commit ef6a851 Jun 4, 2025
3 checks passed
@MauroToscano MauroToscano deleted the 1728-hotfixoperator-manage-subscription-to-events-without-panic branch June 4, 2025 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

hotfix(operator): manage subscription to events without panic
5 participants