Allow the --beacon-nodes list to be updated at runtime

[macladson]

Proposed Changes

Adds a new /lighthouse API call to the VC which allows the list of beacon nodes to be updated dynamically at runtime.

An entirely new beacon node list is provided to the VC so it effectively adds, removes or reorders nodes to match the new list.

This can then be used in Siren, which will enable a "drag to reorder" system along with adding and removing beacon nodes while the VC is on. This will make it unnecessary to reboot the VC when a users want to simply add or remove a BN from the list.

Usage

curl -X POST http://127.0.0.1:5062/lighthouse/beacon/update \
 -H "Authorization: Bearer $(cat ${DATADIR}/validators/api-token.txt)" \
 -H "Content-Type: application/json" \
 -d "{\"beacon_nodes\":[\"http://beacon-node1:5052\",\"http://beacon-node2:5052\",\"http://beacon-node3:5052\"]}" | jq

The new nodes should start appearing in the VC logs, but you can also check which nodes the VC is connected to using:

curl -X GET http://127.0.0.1:5062/lighthouse/ui/fallback_health -H "Authorization: Bearer $(cat ${DATADIR}/validators/api-token.txt)" | jq

Additional Information

The current design rebuilds the candidate list from scratch and then swaps it in. While I don't expect this process to take a long time, it's possible some API queries will fail during the swap over. Another option is to mutate the list in place, which prevents BNs which are present in both the old and new lists from being removed.

[macladson]

Just noting that adding or removing proposer nodes is currently not possible.
The main reason I didn't include it is because if no proposer nodes are included when the VC is initialized, the proposer_node field in BlockService is set to None. So any future attempts to add proposer nodes will require the entire BlockService to be updated.

I suspect we need to rework the way proposer nodes work anyway. I might look into that for a separate PR.

[chong-he]

Done some testing and it is a nice feature to have! During testing, the order of the beacon nodes are successfully re-ordered (verify by querying the GET health endpoint and it shows the index field is re-ordered).

One thing I wonder is how does the unsuccessful POST query look like? I try to insert a non-existing beacon-node in the data field:

DATADIR=/var/lib/lighthouse_test1
curl -X POST http://localhost:5062/lighthouse/beacon/update \
 -H "Authorization: Bearer $(sudo cat ${DATADIR}/validators/api-token.txt)" \
 -H "Content-Type: application/json" \
 -d "{\"beacon_nodes\":[\"http://localhost:5052\",\"http://localhost:5053\",\"http://not-exist:5054\"]}" | jq

The response is as if it reorders the beacon node (including the non-existing one):

{
  "data": {
    "new_beacon_nodes_list": [
      "http://localhost:5052/",
      "http://localhost:5053/",
      "http://not-exist:5054/"
    ]
  }
}

The VC will throw logs of the non-existing beacon node.

Would it be possible to have a check on the beacon nodes used in the --beacon-nodes flag to check the existence of the beacon node in the POST query? That would be nice as it will prevent incorrect POST query to succeed.

[macladson]

Thanks for the review!

Would it be possible to have a check on the beacon nodes used in the --beacon-nodes flag to check the existence of the beacon node in the POST query? That would be nice as it will prevent incorrect POST query to succeed.

I think liveness checking each beacon node is beyond the scope of this API. So long as each value is a valid URL, the beacon node could become "online" at any time so it should be added to the list. I think it makes sense that it is up to the user to provide beacon nodes which are running, just like it is up to the user to provide beacon nodes to the --beacon-nodes CLI flag.

Note that the API will error in the event a value is not a valid URL, or if the list is empty.

[mrabino1]

Is there a target release number to have this functionality pulled in?

[michaelsproul]

Added to v6.1.0