EIP-4844: Free the blobs #3244

arnetheduck · 2023-02-07T10:12:29Z

This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance.

Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations):

Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers
Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are being downloaded
bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size
gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients

This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block.

Changes compared to the current design include:

BlobsSidecar is split into individual BlobSidecar containers - each container is signed individually by the proposer - the signature is used during gossip validation but later dropped.
KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived
- clients may verify individual blob commitments earlier
more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification
by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip
by-range blob requests are done per-block, so as to simplify historical sync
range and root requests are limited to 128 entries for both blocks and blobs - practically, the current higher limit of 1024 for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.

https://hackmd.io/@arnetheduck/BJiJw012j contains further notes and information.

TODO:

implement cryptography properly (Add KZG multi verify function #3236)
(conversation moved to Blob spam protection mechanism #3261) consider more carefully the case of a proposer publishing two blobs for the same index, one valid and one invalid, to separate peers hoping for a network split due to the seen-once rule:
- we can clear the seen flag, then try to download the correct blob from other peers
- we can propagate both blobs and specify that peers that have seen more than one must drop the block from fork choice
should the blobs be signed in the PROPOSER domain or a separate one? (added domain)

This PR reintroduces and further decouples blocks and blobs in EIP-4844, so as to improve network and processing performance. Block and blob processing, for the purpose of gossip validation, are independent: they can both be propagated and gossip-validated in parallel - the decoupled design allows 4 important optimizations (or, if you are so inclined, removes 4 unnecessary pessimizations): * Blocks and blobs travel on independent meshes allowing for better parallelization and utilization of high-bandwidth peers * Re-broadcasting after validation can start earlier allowing more efficient use of upload bandwidth - blocks for example can be rebroadcast to peers while blobs are still being downloaded * bandwidth-reduction techniques such as per-peer deduplication are more efficient because of the smaller message size * gossip verification happens independently for blocks and blobs, allowing better sharing / use of CPU and I/O resources in clients With growing block sizes and additional blob data to stream, the network streaming time becomes a dominant factor in propagation times - on a 100mbit line, streaming 1mb to 8 peers takes ~1s - this process is repeated for each hop in both incoming and outgoing directions. This design in particular sends each blob on a separate subnet, thus maximising the potential for parallelisation and providing a natural path for growing the number of blobs per block should the network be judged to be able to handle it. Changes compared to the current design include: * `BlobsSidecar` is split into individual `BlobSidecar` containers - each container is signed individually by the proposer * the signature is used during gossip validation but later dropped. * KZG commitment verification is moved out of the gossip pipeline and instead done before fork choice addition, when both block and sidecars have arrived * clients may verify individual blob commitments earlier * more generally and similar to block verification, gossip propagation is performed solely based on trivial consistency checks and proposer signature verification * by-root blob requests are done per-blob, so as to retain the ability to fill in blobs one-by-one assuming clients generally receive blobs from gossip * by-range blob requests are done per-block, so as to simplify historical sync * range and root requests are limited to `128` entries for both blocks and blobs - practically, the current higher limit of `1024` for blocks does not get used and keeping the limits consistent simplifies implementation - with the merge, block sizes have grown significantly and clients generally fetch smaller chunks.

specs/eip4844/fork-choice.md

specs/eip4844/p2p-interface.md

djrtwo

Awesome!

I did a first pass. Looks good, a number of small suggestions and questions

djrtwo · 2023-02-07T17:46:16Z

specs/eip4844/beacon-chain.md

@@ -44,6 +44,7 @@ This upgrade adds blobs to the beacon chain as part of EIP-4844. This is an exte
 | Name | SSZ equivalent | Description |
 | - | - | - |
 | `VersionedHash` | `Bytes32` | |
+| `BlobIndex` | `uint64` | |


Why define this here? Just because we don't have a precedent for defining types in the p2p? Seems a bit more natural to define it next to the BlobSideCar def in p2p or validator

That's where we've defined it for other general "index" types (like the ValidatorIndex) - not an opinion, happy to move

specs/eip4844/p2p-interface.md

djrtwo · 2023-02-07T17:52:50Z

specs/eip4844/fork-choice.md


 ```python
 def is_data_available(slot: Slot, beacon_block_root: Root, blob_kzg_commitments: Sequence[KZGCommitment]) -> bool:
    # `retrieve_blobs_sidecar` is implementation and context dependent, raises an exception if not available.
    # Note: the p2p network does not guarantee sidecar retrieval outside of `MIN_EPOCHS_FOR_BLOBS_SIDECARS_REQUESTS`
-    sidecar = retrieve_blobs_sidecar(slot, beacon_block_root)
+    sidecars = retrieve_blob_sidecars(slot, beacon_block_root)


I don't like that we have two different message deliveries -- either in separate sidecars or in an aggregate sidecar -- which makes this retrieval and validation very context dependent. Did you get it from gossip or did you get it in an aggregate side car via byRange.

My preference would be to make this retrieve_aggregate_sidecar and assume that if you got it from local gossip, you can bundle it into the aggregate structure. But this depends on which type of proof (individual or aggregate) is sent in the individual sidecars

Or simply that we abstract the details here such that these fork-choice functions don't care how the blobs were received

cc: @asn-d6

Or simply that we abstract the details here such that these fork-choice functions don't care how the blobs were received

This would be my preference - the design I'm aiming for is to allow the client to validate some commitments individually and some in aggregate, but still mandating that all have passed some validation, before passing the block to fork choice - we could see the aggregation as a pure optimization (assuming aggregate+verify is faster than multiple-single-verifies) - this is the case for "batch verification" of attestations for example.

Actually, that reminds me: we use "batch" to denote any form of aggregation done for the purpose of performance whereas "aggregate" is done for spec reasons. Named that way, we can treat everything up to here as individual blobs without aggregation, but also expose batched-validation-functions in relevant kzg libraries (again, assuming this makes sense, crypto-wise - I do not have a clear picture of this).

The plan at the moment is to provide individual KZG proofs as described here #3244 (comment) . We will still be able to batch-validate them in an amortized way if you have a bunch of blobs and proofs, but we won't use a special aggregated proof to do so.

specs/eip4844/fork-choice.md

specs/eip4844/p2p-interface.md

specs/eip4844/validator.md

specs/eip4844/p2p-interface.md

SebastianSupreme

Here, I think there's a comma missing.

specs/eip4844/validator.md

specs/eip4844/p2p-interface.md

specs/eip4844/fork-choice.md

specs/eip4844/p2p-interface.md

specs/eip4844/fork-choice.md

specs/eip4844/p2p-interface.md

hwwhww · 2023-02-10T06:36:18Z

@arnetheduck since this PR is under active development and is still a draft, I'm not going to force-push to fix the conflicts now. Let me know if you need me to fix the conflicts.

* separate constant for blob requests * pedantry

…future Deneb merge

* expand sidecar gossip conditions * editing * add spec text for `BlobSidecar` signatures

arnetheduck · 2023-02-10T10:29:14Z

@hwwhww thanks, I pushed a merge commit fixing the renames :)

Nashatyrev · 2023-02-10T13:40:58Z

Here is the simulation results on block + blobs dissemination delays (coupled v.s. decoupled). Decoupled approach definitely wins with that respect:

(source data)

Decoupling also appears to save global network traffic:

(original sheet)

specs/deneb/fork-choice.md

djrtwo

excellent! getting close

specs/deneb/p2p-interface.md

specs/deneb/validator.md

hwwhww · 2023-02-16T13:49:27Z

FYI I'm looking into linter and test errors after fixing conflicts. 👀

hwwhww · 2023-02-16T14:19:57Z

@arnetheduck
I fixed conflicts and tests here: https://github.com/ethereum/consensus-specs/tree/pr3244-fix
CI is green now. Let me know if I can push the commits or if you'd like to cherry-pick it.

@asn-d6 @dankrad, it includes some polynomial-commitments.md fixes in a7e45db

arnetheduck · 2023-02-16T16:31:23Z

@hwwhww excellent, thanks! I've pushed the commits here too

djrtwo

fantastic work. just a couple of minor comments and a request for input from @michaelsproul

planning on merging tomorrow

djrtwo · 2023-02-17T00:27:50Z

specs/deneb/p2p-interface.md

+- _[IGNORE]_ The blob's block's parent (defined by `sidecar.block_parent_root`) has been seen (via both gossip and non-gossip sources) (a client MAY queue blocks for processing once the parent block is retrieved).
+- _[REJECT]_ The proposer signature, `signed_blob_sidecar.signature`, is valid with respect to the `sidecar.proposer_index` pubkey.
+- _[IGNORE]_ The sidecar is the only sidecar with valid signature received for the tuple `(sidecar.slot, sidecar.proposer_index, sidecar.index)`.
+  -- If full verification of the blob fails at a later processing stage, clients MUST clear the blob from this "seen" cache so as to allow a the valid blob to propagate. The next block producer MAY orphan the block if they have observed multiple blobs signed by the proposer for the same "seen" tuple.


I understand now. It's using the proposer boost (and if low attestation weight so they know it will likely work).

Hmm, I'm okay putting this in here but maybe should merge the other orphaning spec.
Also the language probably needs to be "MAY attempt to orphan"

CC: @michaelsproul, the re-org god

It's using the proposer boost

I believe lighthouse calls it proposer reorg which in some ways is the opposite of proposer boost: this does not appear in the spec anywhere but rather is a lighthouse-only feature as of now (afair) in which lighthouse validators penalize nodes that publish late blocks.

The open question here is whether we need the collaboration of other nodes to pull this off: do they have to keep track of the duplicate blobs and "blacklist" the offending block as well or is it enough that the next proposer does so?

Proposer re-orgs rely on the proposer-boost being larger than the weight of the attestations that the previous proposer managed to get (e.g. low because they released block late).

A few things

I don't think this really belongs in the networking spec at all. This is a fork-choice capability and a validator option

The obvious way to map that re-org functionality here is if the duplicate blobs reduce the ability of the block to garner attestations, leaving it re-orgable by proposer-boost weight at the next slot. I'm generally okay with making this into the fc/validator spec

Doing this in a coordinated fashion (rather than just by the leader in clear times that they can) seems much more dangerous and likely to lead to network splits, missed attestations, and other bad things. So I'm not really open to "everyone can try to re-org it" unless we have a very clear how and a clear analysis on the chances, the impact, the potential losses from honest folks, etc.

My preference for this PR would be to remove this line and move this convo into and issue

specs/deneb/p2p-interface.md

djrtwo · 2023-02-17T00:29:25Z

specs/deneb/p2p-interface.md

+- _[IGNORE]_ The blob's block's parent (defined by `sidecar.block_parent_root`) has been seen (via both gossip and non-gossip sources) (a client MAY queue blocks for processing once the parent block is retrieved).
+- _[REJECT]_ The proposer signature, `signed_blob_sidecar.signature`, is valid with respect to the `sidecar.proposer_index` pubkey.
+- _[IGNORE]_ The sidecar is the only sidecar with valid signature received for the tuple `(sidecar.slot, sidecar.proposer_index, sidecar.index)`.
+  -- If full verification of the blob fails at a later processing stage, clients MUST clear the blob from this "seen" cache so as to allow a the valid blob to propagate. The next block producer MAY orphan the block if they have observed multiple blobs signed by the proposer for the same "seen" tuple.


Suggested change

-- If full verification of the blob fails at a later processing stage, clients MUST clear the blob from this "seen" cache so as to allow a the valid blob to propagate. The next block producer MAY orphan the block if they have observed multiple blobs signed by the proposer for the same "seen" tuple.

-- If full verification of the blob fails at a later processing stage, clients MUST clear the blob from this "seen" cache so as to allow a potential valid blob to propagate.

-- The next block producer MAY orphan the block if they have observed multiple blobs signed by the proposer for the same "seen" tuple.

Do we need to put this note about clearing the seen cache? We have the same problem with attestations and blocks today. IIUC this cache and IGNORE condition can slow down propagation, but IWANT/IHAVE and subsequently querying for message dependencies will still allow getting such messages even if the cache here hits.

Curious your perspective on this before merge

We have the same problem with attestations and blocks today.

In the past, we've considered it "risky" enough for the proposer to try to poison this seen cache because slashing conditions.

Now that I think about it, maybe the most simple solution, rather than playing these "first" games, is to remove the condition altogether - we want duplicates to propagate to a certain extent so they reach the validators that are in a position to enforce the rules.

I've opened #3257 which covers blocks and attestations.

The rule here is a bit more nuanced in that there's no actual slashing message to rely on - instead, we would rely on the "softer" enforcement of having the next honest proposer orphan such a history. Having written #3257, I think the best course of action is actually to remove the rule altogether and allow implementations to do spam prevention measures individually, for example by limiting to N observed tuples or IGNORE:ing duplicates outside of a much tighter 4s window.

but IWANT/IHAVE and subsequently querying for message dependencies

IGNORE messages are also not given to the IWANT message cache, they're gone.

I still don't think we should clear the seen cache. In the event of a malicious proposer, message dependencies are the best path forward, imo. Gossip will be much slower for such a dishonest proposer, potentially leading to orphaning, but due to dependencies not a permanent split

note, getting a block from a peer doesn't necessarily mean they have the correct blobs, but getting an attestation or child block from a peer does. So these dependency messages become signals on who to download the blobs from and won't lead to unresolvable partitions.

Additionally, if you don't get any dependency messages, then the block will be orphaned. which was at the direct cause of the proposer trying to dos the blobs topic.

So I'd say, keep the cache.

My preference is to keep the cache as we do in other message types and to take this to an issue for discussion. Instead of blocking on this to merge the bulk of the PR

54d2559 - good point about the block likely being orphaned anyway - let's hope that is discouragement enough.

asn-d6 · 2023-02-17T08:35:36Z

@arnetheduck I fixed conflicts and tests here: https://github.com/ethereum/consensus-specs/tree/pr3244-fix CI is green now. Let me know if I can push the commits or if you'd like to cherry-pick it.

@asn-d6 @dankrad, it includes some polynomial-commitments.md fixes in a7e45db

LGTM!

specs/deneb/p2p-interface.md

terencechain · 2023-02-17T19:47:07Z

specs/deneb/p2p-interface.md

+- _[IGNORE]_ The sidecar is from a slot greater than the latest finalized slot -- i.e. validate that `sidecar.slot > compute_start_slot_at_epoch(state.finalized_checkpoint.epoch)`
+- _[IGNORE]_ The blob's block's parent (defined by `sidecar.block_parent_root`) has been seen (via both gossip and non-gossip sources) (a client MAY queue blocks for processing once the parent block is retrieved).
+- _[REJECT]_ The proposer signature, `signed_blob_sidecar.signature`, is valid with respect to the `sidecar.proposer_index` pubkey.
+- _[IGNORE]_ The sidecar is the only sidecar with valid signature received for the tuple `(sidecar.slot, sidecar.proposer_index, sidecar.index)`.


Implementing this, any thoughts on using (sidecar.block_root, sidecar.index). This is an implementation detail but I find that nicer to be able to share the same cache for serving BlobIdentifier requests.

EDIT: this won't work on equivocation for blocks with different contents

EDIT: this won't work on equivocation for blocks with different contents

this would be slashable, if done by the same proposer, so it seems fine - or is there another case you're thinking of?

* also, use root/index for uniqueness

specs/deneb/p2p-interface.md

Co-authored-by: g11tech <develop@g11tech.io>

sama as block

djrtwo

🔥

arnetheduck marked this pull request as draft February 7, 2023 10:12

arnetheduck added 2 commits February 7, 2023 11:14

doctoc

1e07685

fix member

deb82e2

arnetheduck changed the title ~~Free the blobs~~ EIP-4844: Free the blobs Feb 7, 2023

sauliusgrigaitis reviewed Feb 7, 2023

View reviewed changes

specs/eip4844/fork-choice.md Outdated Show resolved Hide resolved

terencechain mentioned this pull request Feb 7, 2023

EIP-4844 Implementers' Call #14 ethereum/pm#718

Closed

mehdi-aouadi mentioned this pull request Feb 7, 2023

Implement EIP-4844 (proto danksharding) Consensys/teku#5681

Closed

58 tasks

terencechain reviewed Feb 7, 2023

View reviewed changes

djrtwo reviewed Feb 7, 2023

View reviewed changes

dapplion reviewed Feb 8, 2023

View reviewed changes

specs/eip4844/validator.md Outdated Show resolved Hide resolved

dapplion reviewed Feb 8, 2023

View reviewed changes

specs/eip4844/p2p-interface.md Outdated Show resolved Hide resolved

pawanjay176 mentioned this pull request Feb 8, 2023

Decouple gossip sigp/lighthouse#3950

Closed

This comment was marked as outdated.

Sign in to view

This comment was marked as resolved.

Sign in to view

SebastianSupreme reviewed Feb 8, 2023

View reviewed changes

specs/eip4844/validator.md Outdated Show resolved Hide resolved

realbigsean reviewed Feb 8, 2023

View reviewed changes

specs/eip4844/p2p-interface.md Outdated Show resolved Hide resolved

asn-d6 mentioned this pull request Feb 8, 2023

Interface changes to free the blobs ethereum/c-kzg-4844#123

Merged

hwwhww added the Deneb was called: eip-4844 label Feb 9, 2023

hwwhww reviewed Feb 10, 2023

View reviewed changes

arnetheduck added 3 commits February 10, 2023 10:40

fixes

ffc78e9

* separate constant for blob requests * pedantry

Merge remote-tracking branch 'origin/dev' into back-to-the-decoupled-…

20278d4

…future Deneb merge

fixes

8bc19d9

* expand sidecar gossip conditions * editing * add spec text for `BlobSidecar` signatures

jimmygchen mentioned this pull request Feb 13, 2023

Add blob download endpoint (getBlobs) ethereum/beacon-APIs#286

Merged

djrtwo reviewed Feb 13, 2023

View reviewed changes

specs/deneb/fork-choice.md Outdated Show resolved Hide resolved

djrtwo reviewed Feb 13, 2023

View reviewed changes

jimmygchen reviewed Feb 14, 2023

View reviewed changes

specs/deneb/validator.md Show resolved Hide resolved

jimmygchen mentioned this pull request Feb 16, 2023

Add Deneb (EIP-4844) types and presets ralexstokes/ethereum-consensus#170

Merged

hwwhww added 2 commits February 16, 2023 21:07

Merge branch 'dev' into pr3244

c2315c9

Make linter happy

f23ed0c

Fix verify_kzg_proof_batch and the tests

a7e45db

realbigsean mentioned this pull request Feb 16, 2023

Blob decoupling sigp/lighthouse#3989

Closed

10 tasks

StefanBratanov mentioned this pull request Feb 16, 2023

Implement Blob Decoupling Consensys/teku#6822

Closed

44 tasks

djrtwo reviewed Feb 17, 2023

View reviewed changes

mehdi-aouadi reviewed Feb 17, 2023

View reviewed changes

specs/deneb/p2p-interface.md Outdated Show resolved Hide resolved

fix Blob pluralization in a few places

9dd7d2b

terencechain reviewed Feb 17, 2023

View reviewed changes

remove producer reorg on multi-blob

54d2559

* also, use root/index for uniqueness

arnetheduck mentioned this pull request Feb 18, 2023

Blob spam protection mechanism #3261

Open

g11tech reviewed Feb 20, 2023

View reviewed changes

specs/deneb/p2p-interface.md Outdated Show resolved Hide resolved

Update specs/deneb/p2p-interface.md

4e2a992

Co-authored-by: g11tech <develop@g11tech.io>

g11tech mentioned this pull request Feb 20, 2023

feat: free the blobs ChainSafe/lodestar#5181

Merged

26 tasks

add parent validation requirement

ac0ec66

sama as block

djrtwo approved these changes Feb 20, 2023

View reviewed changes

djrtwo merged commit ad58bfc into ethereum:dev Feb 20, 2023

hwwhww mentioned this pull request Feb 20, 2023

Uncouple BeaconBlockAndBlobsSidecarByRoot #3139

Closed

tersec mentioned this pull request Feb 21, 2023

use consensus spec v1.3.0-rc.3 test vectors status-im/nimbus-eth2#4645

Merged

timbeiko mentioned this pull request Feb 21, 2023

EIP-4844 Implementers' Call #16 ethereum/pm#726

Closed

g11tech mentioned this pull request Mar 4, 2023

Update EIP-4844: Decouple blobs ethereum/EIPs#6610

Merged

jimmygchen mentioned this pull request Mar 13, 2023

Update KZG utils to support decoupled blobs sigp/lighthouse#4076

Closed

g11tech mentioned this pull request Apr 15, 2023

Deneb tracker ChainSafe/lodestar#5279

Closed

40 tasks

arnetheduck mentioned this pull request Oct 27, 2023

Consider Blob Sidecar slashing conditions #3527

Open

mkalinin mentioned this pull request Oct 31, 2023

Add blob sidecar inclusion proof #3531

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EIP-4844: Free the blobs #3244

EIP-4844: Free the blobs #3244

arnetheduck commented Feb 7, 2023 •

edited

Loading

djrtwo left a comment

djrtwo Feb 7, 2023

arnetheduck Feb 10, 2023

djrtwo Feb 7, 2023

arnetheduck Feb 10, 2023

asn-d6 Feb 13, 2023

This comment was marked as outdated.

This comment was marked as resolved.

SebastianSupreme left a comment

hwwhww commented Feb 10, 2023

arnetheduck commented Feb 10, 2023

Nashatyrev commented Feb 10, 2023

djrtwo left a comment

hwwhww commented Feb 16, 2023

hwwhww commented Feb 16, 2023 •

edited

Loading

arnetheduck commented Feb 16, 2023

djrtwo left a comment

djrtwo Feb 17, 2023

arnetheduck Feb 17, 2023

djrtwo Feb 17, 2023

arnetheduck Feb 18, 2023

djrtwo Feb 17, 2023

djrtwo Feb 17, 2023

arnetheduck Feb 17, 2023

arnetheduck Feb 17, 2023

djrtwo Feb 17, 2023

djrtwo Feb 17, 2023

arnetheduck Feb 18, 2023

asn-d6 commented Feb 17, 2023

terencechain Feb 17, 2023 •

edited

Loading

arnetheduck Feb 20, 2023

djrtwo left a comment

	-- If full verification of the blob fails at a later processing stage, clients MUST clear the blob from this "seen" cache so as to allow a the valid blob to propagate. The next block producer MAY orphan the block if they have observed multiple blobs signed by the proposer for the same "seen" tuple.
	-- If full verification of the blob fails at a later processing stage, clients MUST clear the blob from this "seen" cache so as to allow a potential valid blob to propagate.
	-- The next block producer MAY orphan the block if they have observed multiple blobs signed by the proposer for the same "seen" tuple.

EIP-4844: Free the blobs #3244

EIP-4844: Free the blobs #3244

Conversation

arnetheduck commented Feb 7, 2023 • edited Loading

djrtwo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as resolved.

SebastianSupreme left a comment

Choose a reason for hiding this comment

hwwhww commented Feb 10, 2023

arnetheduck commented Feb 10, 2023

Nashatyrev commented Feb 10, 2023

djrtwo left a comment

Choose a reason for hiding this comment

hwwhww commented Feb 16, 2023

hwwhww commented Feb 16, 2023 • edited Loading

arnetheduck commented Feb 16, 2023

djrtwo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asn-d6 commented Feb 17, 2023

terencechain Feb 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djrtwo left a comment

Choose a reason for hiding this comment

arnetheduck commented Feb 7, 2023 •

edited

Loading

hwwhww commented Feb 16, 2023 •

edited

Loading

terencechain Feb 17, 2023 •

edited

Loading