Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement v2 client GET functionality #972

Merged
merged 45 commits into from
Jan 24, 2025
Rate limit · GitHub

Whoa there!

You have triggered an abuse detection mechanism.

Please wait a few minutes before you try again;
in some cases this may take up to an hour.

Merged
Changes from 1 commit
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
4625e96
Write GET tests
litt3 Dec 9, 2024
f07e820
Merge branch 'master' into client-v2-get
litt3 Dec 10, 2024
885c131
Respond to PR comments
litt3 Dec 10, 2024
6848663
Create new V2 client config
litt3 Dec 10, 2024
a48afb1
Respond to more PR comments
litt3 Dec 11, 2024
225f2a3
Fix failing unit test
litt3 Dec 12, 2024
d265f6a
Merge branch 'master' into client-v2-get
litt3 Dec 12, 2024
e9d91c5
Adopt new package structure
litt3 Dec 12, 2024
dd3c262
Use new test random util
litt3 Dec 12, 2024
88df865
Implement relay call timeout
litt3 Dec 12, 2024
505a1f0
Use correct error join method
litt3 Dec 12, 2024
2b87633
Merge branch 'master' into client-v2-get
litt3 Jan 8, 2025
cf1cd80
Make updates required by upstream changes
litt3 Jan 8, 2025
53893d8
Update how FFT and IFFT are referred to
litt3 Jan 13, 2025
0373dd7
Implement GetPayload
litt3 Jan 13, 2025
826a026
Remove GetBlob, leaving only GetPayload
litt3 Jan 13, 2025
975b6e5
Remove unnecessary codec mock
litt3 Jan 13, 2025
0666d24
Use more reasonable line breaks for logs
litt3 Jan 13, 2025
0a49aa5
Test malicious cert
litt3 Jan 13, 2025
1193ce7
Merge branch 'master' into client-v2-get
litt3 Jan 13, 2025
496e277
Merge branch 'master' into client-v2-get
litt3 Jan 14, 2025
2d392ff
Finish test coverage
litt3 Jan 14, 2025
db51291
Fix commitment length check
litt3 Jan 14, 2025
4f3280c
Merge branch 'master' into client-v2-get
litt3 Jan 16, 2025
aaa1342
Call VerifyBlobV2
litt3 Jan 17, 2025
9be51e6
Simply verify blob
litt3 Jan 17, 2025
cc6b9a1
Merge branch 'master' into client-v2-get
litt3 Jan 17, 2025
ae926c7
Clean up
litt3 Jan 17, 2025
f82d128
Merge branch 'master' into client-v2-get
litt3 Jan 17, 2025
017a48c
Return error from verification method
litt3 Jan 21, 2025
b645370
Merge branch 'master' into client-v2-get
litt3 Jan 21, 2025
03f8018
Address some PR comments
litt3 Jan 21, 2025
ef3944d
Rename methods, and clean up
litt3 Jan 21, 2025
78cab0d
Actually apply fix for poor doc
litt3 Jan 22, 2025
e27d3ea
Fix goroutine safety comment
litt3 Jan 22, 2025
f6126af
Merge branch 'master' into client-v2-get
litt3 Jan 22, 2025
28c3d02
Fix test
litt3 Jan 22, 2025
036a222
Rework polynomial encoding enum, and descriptions
litt3 Jan 22, 2025
7b66df6
Make PR fixes
litt3 Jan 23, 2025
ad3dc97
Move conversion utils
litt3 Jan 23, 2025
6930a47
Remove GetCodec
litt3 Jan 23, 2025
ec190ca
Merge branch 'master' into client-v2-get
litt3 Jan 24, 2025
d27c463
Merge branch 'master' into client-v2-get
litt3 Jan 24, 2025
840ca9a
Fix merge
litt3 Jan 24, 2025
16f0c74
Add additional comment about random
litt3 Jan 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use more reasonable line breaks for logs
Signed-off-by: litt3 <102969658+litt3@users.noreply.github.com>
Rate limit · GitHub

Whoa there!

You have triggered an abuse detection mechanism.

Please wait a few minutes before you try again;
in some cases this may take up to an hour.

litt3 committed Jan 13, 2025
commit 0666d240099ed9c21a1726f4f4c6d155e36569d1
29 changes: 6 additions & 23 deletions api/clients/v2/eigenda_client.go
Original file line number Diff line number Diff line change
@@ -13,7 +13,7 @@ import (
"github.com/consensys/gnark-crypto/ecc/bn254"
)

// EigenDAClient provides the ability to get blobs from the relay subsystem, and to send new blobs to the disperser.
// EigenDAClient provides the ability to get payloads from the relay subsystem, and to send new payloads to the disperser.
//
// This struct is not threadsafe.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is it not goroutine safe? Pretty sure this is used in the proxy which can receive parallel requests that are treated in separate goroutines. Do we have a race condition right now?

Copy link
Contributor Author

@litt3 litt3 Jan 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This client wraps a RelayClient, which doesn't specify thread safety. So I started with "not threadsafe", for the sake of caution

But more fundamentally, my question would be if the proxy actually needs parallelism here. The requests are treated in separate goroutines, but afaict they are being served by a single Manager, which could synchronize access to this client.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't want to synchronize access to the client in proxy though! Client requests can take a long time to deal with because of multiple network routines. Plus when dispersing you need to wait for the CERTIFIED status to be ready which can take up to 10 seconds.

We should be much more fine-grained with synchronization. I would suggest we put a mutex around rand (would say that we don't even care if theres a race on this source of randomness, but perhaps it can actually cause some weird stuff to happen...?).

Also seems like accountant in DisperserClient is thread safe (see this discussion) so I think we should be fine for concurrent support with these 2, but let's do a quick check to make sure there's nothing else we're missing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed synchronously, there doesn't appear to be anything that would make RelayClient unsafe.

We also discussed that synchronizing in the proxy is not acceptable, due to very long times for dispersal (~10 seconds)

I've updated this comment accordingly e27d3eaa

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait but its not! Missing mutex around rand

Copy link
Contributor Author

@litt3 litt3 Jan 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capturing our synchronous discussion

  • the way that random is currently being used in this PR is goroutine safe
  • not all methods on Rand are guaranteed safe

I added an additional comment about this 16f0c748

type EigenDAClient struct {
@@ -137,29 +137,18 @@ func (c *EigenDAClient) verifyBlobFromRelay(

// TODO: in the future, this will be optimized to use fiat shamir transformation for verification, rather than
// regenerating the commitment: https://github.com/Layr-Labs/eigenda/issues/1037
valid, err := verification.GenerateAndCompareBlobCommitment(
c.g1Srs,
blob,
blobCommitments.Commitment)
valid, err := verification.GenerateAndCompareBlobCommitment(c.g1Srs, blob, blobCommitments.Commitment)
if err != nil {
c.log.Warn(
"error generating commitment from received blob",
"blobKey",
blobKey,
"relayKey",
relayKey,
"error",
err)
"blobKey", blobKey, "relayKey", relayKey, "error", err)
return false
}

if !valid {
c.log.Warn(
"blob commitment is invalid for received bytes",
"blobKey",
blobKey,
"relayKey",
relayKey)
"blobKey", blobKey, "relayKey", relayKey)
return false
}

@@ -168,14 +157,8 @@ func (c *EigenDAClient) verifyBlobFromRelay(
if uint(len(blob)) != blobCommitments.Length {
c.log.Warn(
"blob length doesn't match length claimed in blob commitments",
"blobKey",
blobKey,
"relayKey",
relayKey,
"blobLength",
len(blob),
"blobCommitments.Length",
blobCommitments.Length)
"blobKey", blobKey, "relayKey", relayKey, "blobLength", len(blob),
"blobCommitments.Length", blobCommitments.Length)
return false
}

Rate limit · GitHub

Whoa there!

You have triggered an abuse detection mechanism.

Please wait a few minutes before you try again;
in some cases this may take up to an hour.