[Node Issue]: Archive nodes disk usage

[cbolden15]

Contact Details

bd-near slack channel or cbolden@blockdaemon.com

What happened?

Near archive nodes are growing at a rate not seen prior to the implementation of split storage mode. During the month of March we saw an average increase of 4tb of disk space. This level of growth is unsustainable as operators will not be able to afford to run archive nodes in the future.

Version

# neard --version
neard (release 1.38.0) (build unknown) (rustc 1.75.0) (protocol 65) (db 38)

Node type

Split Storage Archival

Are you a validator?

• I am a validator.

Relevant log output

26.816Ti [##########] /2024-04-03T03:17:45Z
   26.934Ti [##########] /2024-03-31T00:07:43Z
   26.301Ti [##########] /2024-03-28T00:07:36Z
   26.136Ti [##########] /2024-03-27T04:07:51Z
   25.890Ti [##########] /2024-03-26T00:07:45Z
   25.837Ti [##########] /2024-03-25T00:07:34Z
   25.704Ti [##########] /2024-03-24T00:07:24Z
   25.449Ti [######### ] /2024-03-23T00:07:42Z
   25.283Ti [######### ] /2024-03-22T03:20:09Z
   24.964Ti [######### ] /2024-03-20T00:07:17Z
   24.857Ti [######### ] /2024-03-19T00:07:41Z
   24.730Ti [######### ] /2024-03-18T00:07:34Z

Node head info

docker container run --rm --name near-d -v /data/near:/root/.near -e RUST_LOG=warn nearprotocol/nearcore:1.38.0 neard --home /root/.near --unsafe-fast-startup view-state scan-db-column --column BlockMisc
2024-04-03T14:38:07.701751Z  WARN neard: /root/.near/config.json: encountered unrecognised fields: store.state_snapshot_config.compaction_enabled, store.state_snapshot_compaction_enabled, cold_store.?.state_snapshot_config.compaction_enabled, cold_store.?.state_snapshot_compaction_enabled
2024-04-03T14:38:07.705206Z  WARN genesis: Skipped genesis validation
2024-04-03T14:38:07.705224Z  WARN genesis: Skipped genesis validation
"CHUNK_TAIL": 115876306
"COLD_HEAD": Tip { height: 116002442, last_block_hash: H4Uj4rxqCwoFWbRk1TconqLkLkzePyPGLtMCEB1t989K, prev_block_hash: BNS52Gp9GqgPxbfG8Uo2hTZe5pYLAgPJiok6aL676kbV, epoch_id: EpochId(EVcdmjn9GWdDusCGjCJ27waXFwtBNdSMaRHvZoXt1VAG), next_epoch_id: EpochId(FcXMkELZvaCi6L5YUL6PWfW462NCwv7mrVZPBQMtQQB4) }
"FINAL_HEAD": Tip { height: 116002446, last_block_hash: 6vfGWKaMzhQv7xmXJsNzvobMb66dgp1JYN9jKwzurLcz, prev_block_hash: 2R7H19U6V5K5PomH2kGchJcmJ4aWf7FE1LDgTx8aMudH, epoch_id: EpochId(EVcdmjn9GWdDusCGjCJ27waXFwtBNdSMaRHvZoXt1VAG), next_epoch_id: EpochId(FcXMkELZvaCi6L5YUL6PWfW462NCwv7mrVZPBQMtQQB4) }
"FORK_TAIL": 115876307
"GENESIS_JSON_HASH": 93on1kcuqTXU94zGyGvBm3YYpPqCkaM8bssbxndgbeRX
"GENESIS_STATE_ROOTS": [8EhZRfDTYujfZoUZtZ3eSMB9gJyFo5zjscR12dEcaxGU]
"HEAD": Tip { height: 116002448, last_block_hash: ApyA7YvHmQbWUqeXyWyZ1unYFzsAu7KJjoizsCGMXgGv, prev_block_hash: 3UH9SXMrpfUPe9U3w5XfoEo7yMXUCSKMfCWYjdv1LogD, epoch_id: EpochId(EVcdmjn9GWdDusCGjCJ27waXFwtBNdSMaRHvZoXt1VAG), next_epoch_id: EpochId(FcXMkELZvaCi6L5YUL6PWfW462NCwv7mrVZPBQMtQQB4) }
"HEADER_HEAD": Tip { height: 116053584, last_block_hash: 2UnAKCjhaiCVoWTFChTg3rw4Hr53i7Np8pyq9KwcXcae, prev_block_hash: Dfc7AMVkMbhnbXvSE2jty961ZvWf8wCgsLtYy9kBVcsr, epoch_id: EpochId(CcGs4B7N4RKnwX7S3aEMzVWpv46Nmdtjb5jBnJkfaGsB), next_epoch_id: EpochId(DEiokvsj2jJHBGiKNHzKmQSaC7kdBQZVgZAE6jD2NAhn) }
"LARGEST_TARGET_HEIGHT": 116002448
"LATEST_KNOWN": LatestKnown { height: 116053584, seen: 1712155074078217846 }
"SYNC_HEAD": Tip { height: 13740748, last_block_hash: 69A1wh25GwoD2CzEuQhs8D2goWPXqe1Liu3jq1i5tdMS, prev_block_hash: 8ZdbgiXn3JpfGGdMGByMqVE5GppKYrojjHJjgZajNV8Z, epoch_id: EpochId(EeWh36LxiVaZgQRsyCzyAUBhaL5yACKZSjK2vTobAC4d), next_epoch_id: EpochId(7edSVzdsSoo1ujdy79abYv3ztbfx7WDawdhCgKhK5qjj) }
"TAIL": 115876307

Node upgrade history

Node created 15 days ago, upgraded from 1.37.1 to 1.38.0

DB reset history

Synced from snapshot about 15 days ago, no DB reset since then

[telezhnaya]

Hey!
Sorry of the late response. Could you please check if it's still an issue?