Run standalone gRPC server for serving historical queries

[amaury1093]

Summary

Add a flag simd start --grpc-only which would not run a node, but only the grpc server.

Problem Definition

Serving historical state is complicated after an upgrade. The current preferred solution, discussed in #9740, is a proxy server which sits in front of 2 nodes:

• one stuck at the upgrade height, for serving queries prior to the upgrade
• one keeping up with the tip, serving queries after the upgrade

Currently, it's hard to run the 1st node, because we hardcode a panic at the upgrade height.

Proposal

Add a CLI --grpc-only flag on start command to run the grpc server only. This means that it will not run the node (no Tendermint, no baseapp), just simply start the gRPC server with direct access to the store. We might need to craft a nice sdk.Context to be passed into the gRPC query handlers.

With this proposal, the process would be:

• run the old binary, wait til panic on upgrade height
• run the new binary
• run the old binary again, with --grpc-only
  Cosmovisor can help smoothen the process.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[iramiller]

This is something we are keenly interested in for Provenance.

One extra consideration that should be taken into account is that this implementation should be easy to back port onto older 0.4x.x SDK based releases. I expect that Provenance will not be the only one looking to patch previous releases so they can be used against prior block height ranges.

[aaronc]

Yes we need this internally for Regen as well.

[alexanderbez]

ACK. Great idea and probably would be very trivial to implement. I can take this on if you'd like.

[amaury1093]

BTW we also need to copy the ~/.appd/data folder, right? Because the new binary will read (and modify) the same folder. We want the old binary to read from a snapshot/copy of that folder at upgrade height.

[andynog]

BTW we also need to copy the ~/.appd/data folder, right? Because the new binary will read (and modify) the same folder. We want the old binary to read from a snapshot/copy of that folder at upgrade height.

Does this mean archive nodes will need 2x the amount of storage at least ? (at upgrade time).

Or would be possible to have a pruning that has a height parameter that you specify the height of the upgrade and everything before that is pruned but not after. This way you probably won't need double storage but not sure if this would have other implications.

[alexanderbez]

The older node would only need to have heights up to and including the upgrade height. In fact, it can have any heights pruned up until the upgrade height, but then it wouldn't be an archive node.

I'm not sure, but it could be 2x the data. However, if the new node has all heights pruned up til the upgrade height, it would not be 2x the data.

[alexanderbez]

I'm going to work on this feature.

[alexanderbez]

So from my preliminary experiments:

1. I'm not sure we actually need to do anything. Even if the node panics (intentionally), you can still query gRPC and API routes. I've tried this successfully by inducing a manual panic in a random module's BeginBlocker.
2. Even if (1) were not true, implementing grpc-only mode would be difficult without significant rewiring in Tendermint. This is because due to simply calling Tendermint's node.New(), it will attempt to replay blocks which naturally calls ABCI methods, which will again result in the same panic.

[ljk662]

I'm not sure we actually need to do anything. Even if the node panics (intentionally), you can still query gRPC and API routes.

Could you elaborate on how we can do this? When I reach an upgrade height (and/or set --halt-height), the node panics and the process completely shuts down. When the process is not running I can't query any routes.

[alexanderbez]

I'm taking a further look at this to see if there's another approach.