🔷 [ProjectTracking] Checklist for GCS-based State Sync

[Sarah-NEAR]

Goals

Background

State Sync equips validator nodes with State data they need in order to produce blocks. Without it, nodes need to get the state data from outside the chain (e.g. from an S3 snapshot) and constantly spend effort to keep the state up to date with the chain. GCS-based State Sync is the first stage of the effort, relying on GCS as a shared medium to transfer state data between nodes.

Why should NEAR One work on this

State Sync unblocks two features:

• It allows nodes to track a subset of the shards (e.g. chunk-only producers can track a single shard). Since nodes are able to get the state data they need from the network, they no longer required to track all of it locally. Tracking a single shard is one of the main pillars of chain scalability.
• It allows validators and RPC operators to spin-up nodes without relying on the S3 snapshots.

What needs to be accomplished

Largely, for GCS-based State Sync we need to generate state parts, collectively upload them to GCS, download them when needed and apply them on the new node.

Main use case

• Chain scalability, via tracking a sigle shard
• Easier spin-up of nodes, without relying on S3 snapshots.

Links to external documentations and discussions

Additional resources will be added here when they become available.

Estimated effort

Most of the effort was already covered with @nikurt, @ppca covering most of the work. Currently we are polishing rough edges unvelied by allowing nodes to track a single shard.

Assumptions

There are no specific assumptions that this project is making.

Pre-requisites

N/A

Out of scope

N/A

Task list:

Tasks

Beta Give feedback

1. [State Sync] Fix known dumping and catchup issues #9572
   Near Core Node +2
   
2. [State Sync] Setup monitoring #9573
   Near Core Node +2
3. [State Sync] Setup garbage collection of state for old epochs #9574
   Near Core Node +2
   
4. [State Sync] Create separate DB for State snapshots #10457
   Near Core Node +2
   
5. 🔷 [ProjectTracking] Enable mainnet validators to track a single shard #10462
   4 of 4
   A-stateless-validation Epic Node project-tracking +4
   

[gmilescu]

2023-11-27

• Code complete
• Code merged into 1.36
• We have known issue: single shard tracking causes nodes to get stuck (due to flat storage API inconsistencies). Bugfix will be included in 1.37
• Alerting and monitoring completed.
• Next steps: update the reference node config file to enable state sync and single shard tracking.

[nikurt]

• Released in 1.36
• Enabled by default
• We have known issue: single shard tracking causes nodes to get stuck (due to flat storage API inconsistencies). Bugfix will be included in 1.37
• Alerting and monitoring completed.

Next steps: Garbage collect state from GCS

[telezhnaya]

Issue with garbage collection is solved
Some more things to do:

• Create separate DB for State snapshots because they take unpredictable amount of space
• Upload headers to GCP to be able to work with stateless validation

[gmilescu]

Actual work on State Sync (GCS-based) is completed, so it makes sense to close this issue, and open a new one that focuses on enabling single-shard tracking.