Skip to content

[Merged by Bors] - Memory usage reduction #1522

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 9 commits into from
Closed

[Merged by Bors] - Memory usage reduction #1522

wants to merge 9 commits into from

Conversation

paulhauner
Copy link
Member

@paulhauner paulhauner commented Aug 15, 2020 •
edited
Loading

Issue Addressed

NA

Proposed Changes

  • Adds a new function to allow getting a state with a bad state root history for attestation verification. This reduces unnecessary tree hashing during attestation processing, which accounted for 23% of memory allocations (by bytes) in a recent heaptrack observation.
  • Don't clone caches on intermediate epoch-boundary states during block processing.
  • Reject blocks that are known to fork choice earlier during gossip processing, instead of waiting until after state has been loaded (this only happens in edge-case).
  • Avoid multiple re-allocations by creating a "forced" exact size iterator.

Additional Info

NA

@paulhauner paulhauner added the work-in-progress PR is a work-in-progress label Aug 15, 2020
@paulhauner paulhauner changed the title Allow for skipping state roots in block replay Memory usage reduction Aug 15, 2020
@paulhauner paulhauner added ready-for-review The code is ready for review and removed work-in-progress PR is a work-in-progress labels Aug 17, 2020
@paulhauner paulhauner marked this pull request as ready for review August 17, 2020 05:37
michaelsproul
michaelsproul requested changes Aug 17, 2020
View reviewed changes
@paulhauner
Copy link
Member Author

Thanks @michaelsproul! All comments addressed :)

michaelsproul
michaelsproul approved these changes Aug 17, 2020
View reviewed changes
Copy link
Member

@michaelsproul michaelsproul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect!

@michaelsproul michaelsproul added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Aug 17, 2020
@paulhauner
Copy link
Member Author

Yayy!

bors r+

bors bot pushed a commit that referenced this pull request Aug 17, 2020
@paulhauner
Memory usage reduction (#1522)
61d5b59 
## Issue Addressed

NA

## Proposed Changes

- Adds a new function to allow getting a state with a bad state root history for attestation verification. This reduces unnecessary tree hashing during attestation processing, which accounted for 23% of memory allocations (by bytes) in a recent `heaptrack` observation.
- Don't clone caches on intermediate epoch-boundary states during block processing.
- Reject blocks that are known to fork choice earlier during gossip processing, instead of waiting until after state has been loaded (this only happens in edge-case).
- Avoid multiple re-allocations by creating a "forced" exact size iterator.

## Additional Info

NA
@bors
Copy link

bors bot commented Aug 17, 2020

Pull request successfully merged into master.

Build succeeded:

@bors bors bot changed the title Memory usage reduction [Merged by Bors] - Memory usage reduction Aug 17, 2020
@bors bors bot closed this Aug 17, 2020
@paulhauner paulhauner deleted the no-hash-replay branch September 18, 2020 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@michaelsproul michaelsproul michaelsproul approved these changes

Assignees
No one assigned
Labels
ready-for-merge This PR is ready to merge.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@paulhauner @michaelsproul