Initial Beacon Chain Light Client Network specs

[ogenev]

• Add Beacon Chain Light Client Network draft specs
• Remove outdated beacon chain specs

Closes #162

[pipermerriam]

looks like a good starting point to me.

[KonradStaniec]

Great job is nice to see it taking shape 👍

I think my main issue for now is that I am still missing big picture here and still did not figured out how to best use this network ? or who is its user actually ?

As I see it currently main users of this network would be actually light clients operating in the network and for this use case probably storing only LightClientUpdate and LightClientBootstrap would be enough i.e someone could configure own light client with option of connecting to this network, and decrease time to reach the most recent headers form initial trusted block i.e instead of requesting those objects standard way, light client could request them from portal network.

Then after reaching most recent beacon chain headers light client could follow standard gossipubsub LightClientFinalityUpdate and LightClientOptimisticUpdate.

Later we could figure out how to best use this gossiped updates in portal context. I see a potential for cooperation here i.e when turning portal network augmentation in light client have a quicker startup time but in exchange it forwards those updates into portal network, when we do something interesting with them.

Is this how other see it?

[KonradStaniec]

when you talk about gossip, you mean our portal home grown gossip (offer/accept) or rather libp2p gossip which is employed in beacon chain ?

Maybe more general question here is should any participant in portal light client netowrk be also paritcipant in gossipbupsub network ?

[ogenev]

when you talk about gossip, you mean our portal home grown gossip (offer/accept) or rather libp2p gossip which is employed in beacon chain ?

Here we mean only the portal offer/accept gossip mechanism.

Maybe more general question here is should any participant in portal light client netowrk be also paritcipant in gossipbupsub network ?

I think the goal here is to eliminate the need for a light client to participate in libp2p network and use the portal network as the only networking layer.

[KonradStaniec]

As I understand both LightClientFinalityUpdate and LightClientOptimisticUpdate are ephemeral objects which are created by full nodes and gossiped to interested light nodes over gossipbupsub so that those can stay in sync. (or am i missing something here ?)

What is main use case for storing them ? (I kind of assume we need to store them to respond queries)

[ogenev]

What is main use case for storing them ? (I kind of assume we need to store them to respond queries)

Yes, those objects are ephemeral and I'm not sure if we need to store them (probably only storing the recent updates).

[ogenev]

As I see it currently main users of this network would be actually light clients operating in the network and for this use case probably storing only LightClientUpdate and LightClientBootstrap would be enough i.e someone could configure own light client with option of connecting to this network, and decrease time to reach the most recent headers form initial trusted block i.e instead of requesting those objects standard way, light client could request them from portal network.

I think the current narrative is that Portal Network can be used as an alternative to the REST and libp2p networking layers for light clients. It is also possible for a light client to be configured in such a way as to use any of the networking layers.

This is one of the main reasons why we are trying to encapsulate all light client data in the Beacon Chain Light Client subnetwork. I imagine a light client that keeps track of the recent block headers, to get all necessary data from Portal Network, without the need for libp2p implementation or REST requests.

Then after reaching most recent beacon chain headers light client could follow standard gossipubsub LightClientFinalityUpdate and LightClientOptimisticUpdate.

I think another reason for including LightClientFinalityUpdate and LightClientOptimisticUpdate in Beacon Chain Light Client Network is that those data objects can be used in the portal History Network to follow the tip of the chain. So a portal client running the history network needs to run also the light client network.

[KonradStaniec]

I think the current narrative is that Portal Network can be used as an alternative to the REST and libp2p networking layers for light clients.

What would be advantage of using this alternative networking layer ?

My main worry is that by design portal gossip (offer-accept) is slower and less reliable that libp2p gossip, so some one following updates only on portal side would have worst experience that on libp2p client. Where on the other hand portal network shine is storing large quantities of relatively static data and finding it in the network.

My current experiences with light client are that once it reaches tip it follows the updates really without any problems. The main issue, is that sometimes it takes long to start. Sometimes its 2min and sometimes its 10min, I still did not debug this with much details, but non deterministic nature of the problem may hint that it is related to finding peers in the network which serves proper light client updates and then syncing up to the tip. And this imo is nice place for portal network to shine.

Ultimately, maybe even storing those updates would not be such problem (if we really manage to use them in history network), but I kind of do not see how portal gossip would be better than libp2p gossip and therefore it hard for me imagine it as alternative networking layer.

One advantage of portal only light client I see, is that whole traffic is udp based, so it maybe easier go around NAT in some environments, but libp2p also has those capabilities, so I am not sure it is enough.

[ogenev]

My main worry is that by design portal gossip (offer-accept) is slower and less reliable that libp2p gossip, so some one following updates only on portal side would have worst experience that on libp2p client

Could you elaborate on why you think libp2p gossip is slower and less reliable than portal gossip? I'm not very familiar with libp2p but looking at their gossipsub protocol. it looks pretty similar to what offer/accept is trying to accomplish in the portal network, The main benefit I see for libp2p is subscribing/unsubscribing for multiple topics and the management of those.

One downside of libp2p is that we may face a similar problem like REST - not enough full nodes that support the light client topics.

Another question is if we don't store LightClientFinalityUpdate and LightClientOptimisticUpdate in a portal subnetwork, how do we will get those objects in the portal history subnetwork to follow the beacon chain headers?

[kdeme]

Another question is if we don't store LightClientFinalityUpdate and LightClientOptimisticUpdate in a portal subnetwork, how do we will get those objects in the portal history subnetwork to follow the beacon chain headers?

Important here is to note what store means. If the intention of this Portal sub-network is to provide what the Beacon Light Client provides but over the Portal network, then only the latest LightClientFinalityUpdate and LightClientOptimisticUpdate should be kept available (see LightClientStore also). Whether this is on disk or in some cache is up to the client (but probably rather a cache). So the intent here is not to "store" each LightClientFinalityUpdate and LightClientOptimisticUpdate that was available in the past on the network (these are also not available on the libp2p network), unless somebody has a good use case for this.

Could you elaborate on why you think libp2p gossip is slower and less reliable than portal gossip?

I think the difference here is that in Portal, through the current way that offers are done (Neighborhood gossip), the content will be offered to the nodes that are closest to the content id of those content items. Meaning that it probably won't spread that well around the whole network. So I think that other nodes that fall out this range, might not receive it and need to request it by rather "polling" for the latest. Not so efficient and I'm not sure if this will be fast enough (and that probably depends on use cases?).
Adjustments could be made here, every node could/should perhaps store these latest updates and instead of NH gossip a more random/broad targeted gossip could probably be done. I think the original headers gossip and tx gossip networks are/were supposed to do something like that too?

Another item is that the range queries for LightClientUpdatesByRange also wouldn't work.
I think we need to set a static count there or don't do a range at all (which would then be slower and not great for syncing).

An idea however might be just to let every node provide all the historical data. It is <6MB assuming only for MIN_EPOCHS_FOR_BLOCK_REQUESTS data is stored. This would then allow for the range queries as is now.

[kdeme]

I think these need to be fixed ranges in order to allow for unique content ids that than match radii of nodes?

Or, nodes could store all data, see bigger comment.

[KonradStaniec]

@kdeme @ogenev what your opinion on having this as:
light_client_update_key = Container(period: uint64)

So each update will have each own content key and content id ?

I think one of the problem with each peer storing everything is the fact that there maybe cases when you will get stuck because closest peer would have less that desirable number of updates. As example:

1. lets say peer closest to local node will have 2 updates
2. now when issuing findContent for all updates we will always receive this 2 updates even though we may need more. It will happen unless this closes peer updates its view (receives more updates) or we get rid of him from our routing table.

Having each update under different key does not have this problem. I think main problem with this approach is that node needs to issue lookup for each key which have bigger latency but:

• its not big a deal as each period lasts 27h so it is pretty long, so even with higher latency we won't hit the case that we can't catch up with the network.
• we can always do few concurrent lookups i.e light client network can have interface getLightClientUpdatesByRange(start_period: uint64, count: uint64) which underneath will translate this query to several concurrent queries and aggregates results. (or stream results as they come)

[kdeme]

I think one of the problem with each peer storing everything is the fact that there maybe cases when you will get stuck because closest peer would have less that desirable number of updates. As example:

Lookups will/must request to more than just 1 neighboring peer. Else, same issue can occur for just other type of data too.
That being said, in the scenario that every peer stores everything, one could just randomly ask peers, similar as we used to do for the "latest" master accumulator (which has been removed now) in the history network.

• its not big a deal as each period lasts 27h so it is pretty long, so even with higher latency we won't hit the case that we can't catch up with the network.

It is not just about just catching up. I think in Portal clients it will be important to know the head of the chain as fast as possible. As other networks / functionality will rely on this. So for a good UX, I think it will be important that this happens fast (what is fast enough? Lets find out..). I'm assuming that in the current libp2p version the max count of 128 is used as much as possible? Could do the test with lower count values.

[KonradStaniec]

One could argue that most important thing to quickly sync up to the tip is to choose bootstrap which is as close to the tip as possible, then whole dance around light client updates is not really that important. (not sure how often those are made)

I'm assuming that in the current libp2p version the max count of 128 is used as much as possible?

yup that is the case. There is check comparing what is current local sync comitee period and what is latest period, and requesting min(128, latest - current_local)

As other networks / functionality will rely on this. So for a good UX, I think it will be important that this happens fast (what is fast enough? Lets find out..).

I kinda agree on UX point, but at this point it is just guessing game. I also think that distributing 20mb of data across DHT is kinda weird. But on the other hand I find model 1 content item <-> 1 content key much more naturally fitting DHT, and a bit more flexible (nothing prevents node to asks for multiple updates concurrently), where having this 1 contentKey <-> all items feels a bit clunky where one need to resolve what does it really mean all items.

Ultimately, one could have both:

• contentKey to ask for all updates
• contentKey to ask for specific update per period.
  Then peer first ask for all updates to some random peers, and after that, if necessary ask for specific updates which weren't provided by those peers.
  This probably complicates implementation a bit, but we have best of two worlds.

[kdeme]

• contentKey to ask for all updates

When I brought up the possible idea of storing all updates (with a purging after updates are older than the required time to store), I didn't mean to also send them all over a request with 1 contentKey.

It was as a solution to be able to do the range requests. So the contentKey would be related to the range requested.

But this is not the only solution of course. We could also just hard set certain ranges that can be requested, e.g fixed start period + fixed range (128?).

If the ranges are required in the first place of course (for good UX). I realize that this depends on how old the bootstrap is.
The question here is how this would work exactly in a real scenario such as a wallet using a Portal client as back-end.

[KonradStaniec]

When I brought up the possible idea of storing all updates (with a purging after updates are older than the required time to store), I didn't mean to also send them all over a request with 1 contentKey.

yea, I think initially I have misunderstood this point 😅

But then having content key as Container(start_period: uint64, count: uint64)) and every node storing everything, we most probably need some custom lookup procedure as content id from such content key does not have any meaning really.

Thats why I proposed in my last comment that maybe it would be valuable, that each node insead of radius (as if we store everything it does not have any value) would broadcast oldest sync comitee period for which it has data available, so that we can lookup for peers and determine which has data from period intersting to us.

The question here is how this would work exactly in a real scenario such as a wallet using a Portal client as back-end.

yup, my current metal model is that user would only need to specify trusted bootstrap and sync form boostrap only if he was offline longer than 5 months, other than that progress would be saved and lc would be restared from most recent sync comittee period.

[ogenev]

Important here is to note what store means. If the intention of this Portal sub-network is to provide what the Beacon Light Client provides but over the Portal network, then only the latest LightClientFinalityUpdate and LightClientOptimisticUpdate should be kept available (see LightClientStore also).

Yeah, I meant provide the latest LightClientFinalityUpdate and LightClientOptimisticUpdate. I'll update the specs to make it clear.

Adjustments could be made here, every node could/should perhaps store these latest updates and instead of NH gossip a more random/broad targeted gossip could probably be done.

This seems reasonable, randomly targeted gossip is what gossipsub is doing and I guess we can have a similar approach, what do you think @pipermerriam ?

An idea however might be just to let every node provide all the historical data. It is <6MB assuming only for MIN_EPOCHS_FOR_BLOCK_REQUESTS data is stored. This would then allow for the range queries as is now.

I like this idea, so if I understand it correctly, the start_period of LightClientUpdatesByRange should be after the Weak Subjectivity Checkpoint?

[ogenev]

Formatted the specs to match the subprotocol template introduced in #168.

[etan-status]

Not sure how well this maps to portal network. May work better to just store one per period, over storing all possible ranges. If latency is a concern, a count value could be fixed.

[etan-status]

Needs to be fork-aware, the individual items may have different types.
See ethereum/beacon-APIs#247

(Same for the other contents, but there it can be solved by simply prefixing the content with the corresponding forkDigest)

[ogenev]

(Same for the other contents, but there it can be solved by simply prefixing the content with the corresponding forkDigest)

Could you elaborate on this? My understanding is that we need to return ForkDigest only for every LightClientUpdate payload, what other contents do you mean?

[etan-status]

LightClientFinalityUpdate / LightClientOptimisticUpdate / LightClientBootstrap should all be fork aware for future proofing.

Concretely, starting with Capella, those structures may be modified to also include the EL block hash. So, there needs to be some indicator for the receiver to know what structure to parse. Whether that is the fork-digest like on libp2p/REST, or just a simple enum with 1=altair, 2=bellatrix, 3=capella or something else doesn't matter. Technically, it is also possible to rely that attested_header.slot is kept at offset 0, but rather keep that flexible; the 1-4 extra bytes don't hurt.

[etan-status]

There is an important difference compared to other portal network data that is specific to light client data, that there can be multiple valid objects with conflicting content keys due to the way how aggregate signatures work.

The is_better_update function should be used when encountering conflicting data to resolve conflicts. This ensures that if conflicting data is encountered, the one with higher participation in the signature takes precedence.

[etan-status]

Note that this also affects propagation, i.e., when encountering a better update than the local one, the better one should be propagated among peers. Maybe the update quality needs to be encoded into the message id for this purpose.

[etan-status]

Keeping just the latest one should be sufficient for Finality / Optimistic updates. Having access to the latest header refers to all previous header through the parent_root chain.

[etan-status]

In presence of forking, this needs the actual sync committees in the keys as well, to uniquely identify the branch.

To uniquely identify a non-finalized sync committee fork, all of period, current_sync_committee and next_sync_committee need to be incorporated, as sync committees may reappear over time.
https://github.com/ethereum/consensus-specs/blob/dev/specs/altair/light-client/full-node.md#create_light_client_update

[etan-status]

Is the mechanism to distinguish between different networks separate? e.g., mainnet vs goerli? How about individual forks? Maybe needs to incorporate fork_digest in the content keys here.

[pipermerriam]

We don't currently have any plans to support different networks. Portal network is end-user facing, and requires a critical mass of nodes to have sufficient capacity to serve the necessary data. Between these two things we A: don't have a compelling use case to motivate support for test networks and B: don't think the network would be reliable for test networks since there would likely not be sufficient nodes to serve the underlying data.

[KonradStaniec]

At some point we will need to consider how we will inject bootstraps into the network.

Currently in Light Client protocol node choses one trusted block hash for which it will accept a bootstrap. If we would go that road, it would meant that when we would offer boostraps to the network most peers would only the one they trust, which would mean there is not a lot of bootstarp objects in the network.

As for now I see two ways to deal with it:

• make each node specify multiple trusted roots
• validate those out of band. (for example by libp2p light client ?)

[ogenev]

We also want to keep only bootstraps within the weak subjectivity period which is ~5 months, so we will need also a way to purge bootstraps for epochs < (current_epoch - MIN_EPOCHS_FOR_BLOCK_REQUESTS).

[pipermerriam]

IIUC this is a coordination issue, where we simply need the clients to "agree" on which blocks we will consider valid for "bootstrapping". Assuming the selection criteria is arbitrary, can we just define something like a rolling window/epoch of ~2 months worth of blocks. This would ensure that we always had at least 2 block hashes that could be used for bootstrapping within the previous 5 month time window?

[KonradStaniec]

Maybe we should be more clear about this part.

Should this be SSZ.serialize(Container(forkDigest ForkDigest, bootstrap: LightlientBootstrap)) or rather
ForkDigest || SSZ.serialize(LightlientBootstrap) ?

|| is usually append operator.

In this case both versions create exactly the same bytes i.e 4 byte digest prepended to ssz serialized LightlientBootstrap, but maybe it is better to be clear here.

[ogenev]

Right, I'm more in favor of using the append operator ||, do you have any preference @kdeme ?

[KonradStaniec]

I personally prefer - ForkDigest || SSZ.serialize(LightlientBootstrap). Imo we gain nothing here packing it into ssz container

[KonradStaniec]

As we are considering all peers storing every thing (up to weak subjectivity boundary), radius thing is not really necessary for this network right ?

Maybe instead, we could publish here other data, like oldest sync comitee period peer has data for ?

As different peers will start from different point in time and from different boostrap, so the can have light client updates from different periods.
For example:
If current sync committee period is 500, and one peer started from bootstrap at 480, and another at 450, the new peer starting at 460 should probably ask the second one for updates, not the first on as the first one does not have updates from periods 460-480.

[pipermerriam]

If we aren't using radius in this network, I would like us to know what the upper bounds on total data storage will be like for nodes, as well as some numbers on expected bandwidth usage. We need to be sure that we're not introducing anything into client requirements that is going to violate our philosophy of nodes being able to tune their network parameters to match their storage/CPU/bandwidth needs.

[KonradStaniec]

Conteiners without fields not allowed in ssz - https://github.com/ethereum/consensus-specs/blob/dev/ssz/simple-serialize.md#illegal-types.

Maybe, empty list -List[byte, 0] or Container(latest: bool) (although this suggest there is request for not latest update) or just uint8 (with whatever value) ?

[kdeme]

Or, while that might sound confusing, just empty.

It would be the equivalent of Union[None] in case you work with SSZ Union for the selectors.

[pipermerriam]

I think that we should be a little careful here.

First, if this results in a static content-id and this object is being regularly fetched from nodes in the network, then we end up with a hot-spot where nodes that live close to that spot will shoulder most of the responsibility for serving the data.

Second, and likely minor but maybe worth mentioning, if we use any "short" value here, we set ourselves up with a situation where we have to be careful not to map any other "short" values to the same location.

One option to solve the hot-spot issue would be to have the key change location on an epoch boundary every N seconds, picking a value of N that doesn't move things too fast or too slow based on what we expect access patterns to look like.

[kdeme]

The LightClientFinalityUpdate and LightClientOptimisticUpdate are ephemeral objects. The libp2p interface only allows for retrieval of the latest of these objects. And they also get gossiped.
So I think the idea is that for the Portal network we do the same and only provide the latest.

With that in mind, I think it can function similarly like we were having the MasterAccumulator for the history network, before it became a static object added at compile time. That is, every node stores them / has them available and additionally, they get gossiped to all. This does require some changes to the way of requesting them and of gossiping.

One option to solve the hot-spot issue would be to have the key change location on an epoch boundary every N seconds, picking a value of N that doesn't move things too fast or too slow based on what we expect access patterns to look like.

That could be an alternative to letting every node store this.
But on the gossip side, it would probably be better to gossip to all as I think that would make it easier for a node that needs to follow the updates.

Second, and likely minor but maybe worth mentioning, if we use any "short" value here, we set ourselves up with a situation where we have to be careful not to map any other "short" values to the same location.

Good point, while the selector is enough differentiator for content within the subnetwork, if the database is shared over different networks, it can be an issue.

[kdeme]

I did just realize that the Union None value should be only on the 0 selector, and also not for two types: https://github.com/ethereum/consensus-specs/blob/dev/ssz/simple-serialize.md#union

[ogenev]

Changed this field to 0 (uint8).

[pipermerriam]

some idle thoughts I had while paging through this. How close are you guys to feeling like this is ready for clients to start implementing?

[pipermerriam]

If we aren't using radius in this network, I would like us to know what the upper bounds on total data storage will be like for nodes, as well as some numbers on expected bandwidth usage. We need to be sure that we're not introducing anything into client requirements that is going to violate our philosophy of nodes being able to tune their network parameters to match their storage/CPU/bandwidth needs.

[pipermerriam]

IIUC this is a coordination issue, where we simply need the clients to "agree" on which blocks we will consider valid for "bootstrapping". Assuming the selection criteria is arbitrary, can we just define something like a rolling window/epoch of ~2 months worth of blocks. This would ensure that we always had at least 2 block hashes that could be used for bootstrapping within the previous 5 month time window?

[carver]

There are several links to Altair specs. With Shanghai approaching, I think we should probably link the docs to Capella (and check how our implementation needs to change).

[etan-status]

Note that the way how ethereum/beacon-specs works, new spec forks automatically inherit a 1:1 copy from the previous spec fork. So, for the purpose of light clients, Capella is a copy of Altair with just a few smaller functions replaced.