[Merged by Bors] - Share reqwest::Client between validators when using Web3Signer
#3335
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
macladson
added
work-in-progress
bors bot
pushed a commit
that referenced
this pull request
Jul 15, 2022
## Proposed Changes Adds some improvements I found when playing around with local testnets in #3335: - When trying to kill processes, do not exit on a failure. (If a node fails to start due to a bug, the PID associated with it no longer exists. When trying to tear down the testnets, an error will be raised when it tries that PID and then will not try any PIDs following it. This change means it will continue and tear down the rest of the network. - When starting the testnet, set `ulimit` to a high number. This allows the VCs to import 1000s of validators without running into limitations.
macladson
added
ready-for-review
michaelsproul
requested changes
Jul 15, 2022
michaelsproul
added
waiting-on-author
macladson
added
ready-for-review
macladson
added
ready-for-review
michaelsproul
approved these changes
Jul 19, 2022
michaelsproul
added
ready-for-merge
|
bors r+ |
bors bot
pushed a commit
that referenced
this pull request
Jul 19, 2022
## Issue Addressed #3302 ## Proposed Changes Move the `reqwest::Client` from being initialized per-validator, to being initialized per distinct Web3Signer. This is done by placing the `Client` into a `HashMap` keyed by the definition of the Web3Signer as specified by the `ValidatorDefintion`. This will allow multiple Web3Signers to be used with a single VC and also maintains backwards compatibility. ## Additional Info This was done to reduce the memory used by the VC when connecting to a Web3Signer. I set up a local testnet using [a custom script](https://github.com/macladson/lighthouse/tree/web3signer-local-test/scripts/local_testnet_web3signer) and ran a VC with 200 validator keys: VC with Web3Signer: - `unstable`: ~200MB - With fix: ~50MB VC with Local Signer: - `unstable`: ~35MB - With fix: ~35MB > I'm seeing some fragmentation with the VC using the Web3Signer, but not when using a local signer (this is most likely due to making lots of http requests and dealing with lots of JSON objects). I tested the above using `MALLOC_ARENA_MAX=1` to try to reduce the fragmentation. Without it, the values are around +50MB for both `unstable` and the fix.
bors
bot
changed the title
reqwest::Client between validators when using Web3Signerreqwest::Client between validators when using Web3Signer
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue Addressed
#3302
Proposed Changes
Move the
reqwest::Clientfrom being initialized per-validator, to being initialized per distinct Web3Signer.This is done by placing the
Clientinto aHashMapkeyed by the definition of the Web3Signer as specified by theValidatorDefintion. This will allow multiple Web3Signers to be used with a single VC and also maintains backwards compatibility.Additional Info
This was done to reduce the memory used by the VC when connecting to a Web3Signer.
I set up a local testnet using a custom script and ran a VC with 200 validator keys:
VC with Web3Signer:
unstable: ~200MBVC with Local Signer:
unstable: ~35MB