Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Trigger level db compactRange #4515

Open
dapplion opened this issue Sep 7, 2022 · 3 comments
Open

Trigger level db compactRange #4515

dapplion opened this issue Sep 7, 2022 · 3 comments
Labels
meta-investigate Issues found that require further investigation and may not have a specific resolution/fix prio-low This is nice to have. scope-ux Issues for CLI UX or general consumer UX.

Comments

@dapplion
Copy link
Contributor

dapplion commented Sep 7, 2022 •
edited
Loading

Due to level-db internals, after a significant amount of data put or del it's necessary to compact that range to realize the size reduction. After an offline discussion with other teams:

  • Should compact all archived states range after the prune of this PR
  • Should compact after completing sync
  • Should consider compacting the DB at most once per 7 days

Given we have not done this before, extensive testing is necessary to:

  • Benchmark CPU and time cost of compactRange on different key ranges, plus how much size is free'ed on average situtations
  • Understand the impact on performance on the beacon node if any. While NodeJS is mostly single thread, I would expect this ops to happen in the C++ worker thread and not mess with validator performance
@dapplion dapplion added prio-medium Resolve this some time soon (tm). scope-ux Issues for CLI UX or general consumer UX. labels Sep 7, 2022
@dapplion
Copy link
Contributor Author

dapplion commented Sep 7, 2022 •
edited
Loading

I've added a new option on a branch dapplion/bump-level...dapplion/test-compact which triggers compact of the entire db on start up with --compactDbOnStartup. I've deployed this branch in all feat2 hosts as of 11:20 CEST Sept 7th

Results for feat2-sm1v-ctvpss

  • node_filesystem_avail_bytes changed from 68GB -> 121GB: Total freed 53GB
  • Total time 13 min
Sep-07 11:27:00.863[]                 info: db compactRange on byte 0/256 allForks_stateArchive...
Sep-07 11:33:00.127[]                 info: db compactRange on byte 1/256 allForks_block...
Sep-07 11:33:02.192[]                 info: db compactRange on byte 2/256 allForks_blockArchive...
Sep-07 11:40:06.486[]                 info: db compactRange on byte 3/256 index_blockArchiveParentRootIndex...
Sep-07 11:40:11.251[]                 info: db compactRange on byte 4/256 index_blockArchiveRootIndex...
Sep-07 11:40:16.204[]                 info: db compactRange on byte 6/256 index_mainChain...
Sep-07 11:40:16.239[]                 info: db compactRange on byte 7/256 index_chainInfo...
Sep-07 11:40:16.273[]                 info: db compactRange on byte 8/256 phase0_eth1Data...
Sep-07 11:40:16.792[]                 info: db compactRange on byte 9/256 index_depositDataRoot...
Sep-07 11:40:16.897[]                 info: db compactRange on byte 12/256 phase0_depositData...
Sep-07 11:40:16.938[]                 info: db compactRange on byte 13/256 phase0_exit...
Sep-07 11:40:16.972[]                 info: db compactRange on byte 14/256 phase0_proposerSlashing...
Sep-07 11:40:17.009[]                 info: db compactRange on byte 15/256 phase0_attesterSlashing...
Sep-07 11:40:17.167[]                 info: db compactRange on byte 19/256 phase0_depositEvent...
Sep-07 11:40:17.593[]                 info: db compactRange on byte 20/256 phase0_slashingProtectionBlockBySlot...
Sep-07 11:40:17.627[]                 info: db compactRange on byte 21/256 phase0_slashingProtectionAttestationByTarget...
Sep-07 11:40:17.657[]                 info: db compactRange on byte 22/256 phase0_slashingProtectionAttestationLowerBound...
Sep-07 11:40:17.696[]                 info: db compactRange on byte 23/256 index_slashingProtectionMinSpanDistance...
Sep-07 11:40:17.735[]                 info: db compactRange on byte 24/256 index_slashingProtectionMaxSpanDistance...
Sep-07 11:40:17.831[]                 info: db compactRange on byte 26/256 index_stateArchiveRootIndex...
Sep-07 11:40:17.966[]                 info: db compactRange on byte 30/256 phase0_preGenesisState...
Sep-07 11:40:17.998[]                 info: db compactRange on byte 31/256 phase0_preGenesisStateLastProcessedBlock...
Sep-07 11:40:18.416[]                 info: db compactRange on byte 41/256 validator_metaData...
Sep-07 11:40:18.456[]                 info: db compactRange on byte 42/256 backfilled_ranges...
Sep-07 11:40:18.793[]                 info: db compactRange on byte 51/256 lightClient_syncCommitteeWitness...
Sep-07 11:40:19.391[]                 info: db compactRange on byte 52/256 lightClient_syncCommittee...
Sep-07 11:40:19.419[]                 info: db compactRange on byte 53/256 lightClient_checkpointHeader...
Sep-07 11:40:19.813[]                 info: db compactRange on byte 54/256 lightClient_bestPartialLightClientUpdate...
Sep-07 11:40:19.844[]                 info: db compactRange on byte 55/256 -...
...
Sep-07 11:40:27.167[]                 info: db compactRange on byte 254/256 -...
Sep-07 11:40:27.201[]                 info: db compactRange completed

Results for feat2-novc-ctvpss

  • node_filesystem_avail_bytes changed from 68.5GB -> 120GB: Total freed 51.5GB
  • Total time 12 min
devops@feat2-novc-ctvpss:~$ grep compactRange beacon/beacon-2022-09-07.log
Sep-07 11:27:00.795[]                 info: db compactRange on byte 0/256 allForks_stateArchive...
Sep-07 11:33:10.167[]                 info: db compactRange on byte 1/256 allForks_block...
Sep-07 11:33:11.898[]                 info: db compactRange on byte 2/256 allForks_blockArchive...
Sep-07 11:38:18.848[]                 info: db compactRange on byte 3/256 index_blockArchiveParentRootIndex...
Sep-07 11:38:23.226[]                 info: db compactRange on byte 4/256 index_blockArchiveRootIndex...
Sep-07 11:38:26.930[]                 info: db compactRange on byte 6/256 index_mainChain...
Sep-07 11:38:26.966[]                 info: db compactRange on byte 7/256 index_chainInfo...
Sep-07 11:38:27.005[]                 info: db compactRange on byte 8/256 phase0_eth1Data...
Sep-07 11:38:27.822[]                 info: db compactRange on byte 9/256 index_depositDataRoot...
Sep-07 11:38:27.906[]                 info: db compactRange on byte 12/256 phase0_depositData...
Sep-07 11:38:27.939[]                 info: db compactRange on byte 13/256 phase0_exit...
Sep-07 11:38:27.966[]                 info: db compactRange on byte 14/256 phase0_proposerSlashing...
Sep-07 11:38:27.993[]                 info: db compactRange on byte 15/256 phase0_attesterSlashing...
Sep-07 11:38:28.105[]                 info: db compactRange on byte 19/256 phase0_depositEvent...
Sep-07 11:38:28.240[]                 info: db compactRange on byte 20/256 phase0_slashingProtectionBlockBySlot...
Sep-07 11:38:28.267[]                 info: db compactRange on byte 21/256 phase0_slashingProtectionAttestationByTarget...
Sep-07 11:38:28.298[]                 info: db compactRange on byte 22/256 phase0_slashingProtectionAttestationLowerBound...
Sep-07 11:38:28.327[]                 info: db compactRange on byte 23/256 index_slashingProtectionMinSpanDistance...
Sep-07 11:38:28.354[]                 info: db compactRange on byte 24/256 index_slashingProtectionMaxSpanDistance...
Sep-07 11:38:28.419[]                 info: db compactRange on byte 26/256 index_stateArchiveRootIndex...
Sep-07 11:38:28.532[]                 info: db compactRange on byte 30/256 phase0_preGenesisState...
Sep-07 11:38:28.560[]                 info: db compactRange on byte 31/256 phase0_preGenesisStateLastProcessedBlock...
Sep-07 11:38:28.866[]                 info: db compactRange on byte 41/256 validator_metaData...
Sep-07 11:38:28.891[]                 info: db compactRange on byte 42/256 backfilled_ranges...
Sep-07 11:38:29.192[]                 info: db compactRange on byte 51/256 lightClient_syncCommitteeWitness...
Sep-07 11:38:29.825[]                 info: db compactRange on byte 52/256 lightClient_syncCommittee...
Sep-07 11:38:30.121[]                 info: db compactRange on byte 53/256 lightClient_checkpointHeader...
Sep-07 11:38:30.563[]                 info: db compactRange on byte 54/256 lightClient_bestPartialLightClientUpdate...
Sep-07 11:38:30.589[]                 info: db compactRange on byte 55/256 -...
...
Sep-07 11:38:37.320[]                 info: db compactRange on byte 254/256 -...
Sep-07 11:38:37.351[]                 info: db compactRange completed

Results for feat2-novc-ctvpss

Note this instance did not prune additional states beforehand, the two above prune states but did not compact

  • node_filesystem_avail_bytes changed from 128GB -> 128GB: Total freed 0GB
  • Total time 90 min
devops@feat2-md16-ctvpsm:~$ grep compactRange beacon/beacon-2022-09-07.log 
Sep-07 11:27:07.681[]                 info: db compactRange on byte 0/256 allForks_stateArchive...
Sep-07 12:26:58.457[]                 info: db compactRange on byte 0/256 allForks_stateArchive...
Sep-07 12:26:58.519[]                 info: db compactRange on byte 1/256 allForks_block...
Sep-07 12:27:00.476[]                 info: db compactRange on byte 2/256 allForks_blockArchive...
Sep-07 12:56:07.042[]                 info: db compactRange on byte 3/256 index_blockArchiveParentRootIndex...
Sep-07 12:57:24.598[]                 info: db compactRange completed

This was referenced Sep 7, 2022
Merged
Closed
@tmm360
Copy link

tmm360 commented Jul 20, 2023

Any news on this? I'm running lodestar on gnosis, and periodically I have to delete db and redo sync because it is increasing by ~10GB per month.

@dapplion
Copy link
Contributor Author

dapplion commented Jul 21, 2023 •
edited
Loading

Any news on this? I'm running lodestar on gnosis, and periodically I have to delete db and redo sync because it is increasing by ~10GB per month.

This proposal was started to clean-up the consequences of an old bug where Lodestar persisted states too frequently. If you have re-synced Lodestar recently this proposal won't affect much. To reduce disk space we are working on this optimization which should reduce disk requirements significantly. I have to check Gnosis specific numbers, but for Ethereum it could be save 10-30 GB / month

@philknows philknows added meta-investigate Issues found that require further investigation and may not have a specific resolution/fix prio-low This is nice to have. and removed prio-medium Resolve this some time soon (tm). labels Oct 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
meta-investigate Issues found that require further investigation and may not have a specific resolution/fix prio-low This is nice to have. scope-ux Issues for CLI UX or general consumer UX.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tmm360 @dapplion @philknows