validator restart can generate "Cluster largest_confirmed_root x does not exist on node. Node root: y"

[mvines]

When a validator restarts, blockstore_processor filters out all roots older than the current root [1]. By the time it goes to create a BankForks [2], the cluster confirmed root requested at [3] is long gone.

Couple options I see:

1. Introduce ForkProgress into blockstore_processor
2. Update blockstore_processor to hang on to a couple hundred older roots to populate into BankForks [2] (which'll get pruned the first time BankForks::set_root() is called.

[1]

solana/ledger/src/blockstore_processor.rs

Line 747 in 3757754

initial_forks.clear();

[2]

solana/ledger/src/blockstore_processor.rs

Line 381 in 3757754

let bank_forks = BankForks::new_from_banks(&initial_forks, root);

[3]

solana/core/src/rpc.rs

Lines 102 to 113 in 627bc7e

	r_bank_forks
	.get(cluster_root)
	.cloned()
	.ok_or_else(|| Error {
	code: ErrorCode::ServerError(JSON_RPC_SERVER_ERROR_0),
	message: format!(
	"Cluster largest_confirmed_root {} does not exist on node. Node root: {}",
	cluster_root,
	r_bank_forks.root(),
	),
	data: None,
	})

[mvines]

Once this is fixed it's probably fine to back out #10061

[mvines]

This is a problem for http://explorer.solana.com/supply during a manual cluster restart due to:

$ curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc":"2.0", "id":1, "method":"getSupply"}' http://api.mainnet-beta.solana.com
{"jsonrpc":"2.0","error":{"code":-32000,"message":"Cluster largest_confirmed_root 0 does not exist on node. Node root: 13334463"},"id":1}

This unnecessarily causes the explorer to fail with:

and causes unnecessary concern in certain circles