Skip to content

eth/filters: add global block logs cache #25459

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Aug 19, 2022
Merged

Conversation

s1na
Copy link
Contributor

@s1na s1na commented Aug 1, 2022

This adds a cache for block logs which is shared by all filters. In order to share the cache reference, filters now need to be created through a 'filter system' object. Cache size is user-configurable via --cache.blocklogs.

This PR is an alternative to #25425.
cc @ryanschneider

Fixes #25421
Includes #25468, which should be merged first.

@ryanschneider
Copy link
Contributor

lgtm! Thanks!

Copy link
Contributor

@holiman holiman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@@ -266,6 +267,9 @@ func (f *Filter) checkMatches(ctx context.Context, header *types.Header) (logs [
if err != nil {
return nil, err
}
if logsList == nil {
Copy link
Member

@rjl493456442 rjl493456442 Aug 2, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's weird to add this "duplicated" check here, usually a nil error means the returned result is not empty.

I would vote to add the check here https://github.com/ethereum/go-ethereum/pull/25459/files#diff-9886da3412b43831145f62cec6e895eb3613a175b945e5b026543b7463454603R204

func (b *EthAPIBackend) GetLogs(ctx context.Context, hash common.Hash) ([][]*types.Log, error) {
       result := b.eth.blockchain.GetLogsByHash(hash)
       if result == nil {
             return nil, errors.New("xxx")
       }
       return b.eth.blockchain.GetLogsByHash(hash), nil
}

Also in les_apiBackend.GetLogs, we should also return the error if logs is not found.

func (b *LesApiBackend) GetLogs(ctx context.Context, hash common.Hash) ([][]*types.Log, error) {
	if number := rawdb.ReadHeaderNumber(b.eth.chainDb, hash); number != nil {
		return light.GetBlockLogs(ctx, b.eth.odr, hash, *number)
	}
	return nil, fmt.Errorf("failed to get logs for block %x", hash)
}

@fjl
Copy link
Contributor

fjl commented Aug 2, 2022

This is ready for testing now. The cache size is configurable via --cache.logcache or in TOML with Eth.FilterLogCacheSize = 128, for example.

@holiman Please re-check this, the PR has changed significantly after your initial review.

@fjl fjl changed the title core,eth: cache logs in blockchain eth/filters: add global block logs cache Aug 3, 2022
@s1na s1na force-pushed the filter/cache-logs branch from 359a19a to dfb6bd7 Compare August 4, 2022 09:40
JukLee0ira added a commit to JukLee0ira/XDPoSChain that referenced this pull request Jul 23, 2024
JukLee0ira added a commit to JukLee0ira/XDPoSChain that referenced this pull request Aug 3, 2024
gzliudan pushed a commit to XinFinOrg/XDPoSChain that referenced this pull request Aug 3, 2024
wanwiset25 pushed a commit to XinFinOrg/XDPoSChain that referenced this pull request Aug 23, 2024
pdrobnjak pushed a commit to Tenderly/net-scroll-geth that referenced this pull request Apr 30, 2025
…h#42)

* rpc: improve error codes for internal server errors (ethereum#25678)

This changes the error code returned by the RPC server in certain situations:

- handler panic: code -32603
- result marshaling error: code -32603
- attempt to subscribe via HTTP: code -32001

In all of the above cases, the server previously returned the default error
code -32000.

Co-authored-by: Nicholas Zhao <nicholas.zhao@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>

* rpc: add PeerInfo (ethereum#24255)

This replaces the sketchy and undocumented string context keys for HTTP requests
with a defined interface. Using string keys with context is discouraged because
they may clash with keys created by other packages.

We added these keys to make connection metadata available in the signer, so this
change also updates signer/core to use the new PeerInfo API.

* graphql: add query timeout (ethereum#26116)

This PR adds a 60 second timeout to graphql queries.

* graphql, node, rpc: improve HTTP write timeout handling (ethereum#25457)

Here we add special handling for sending an error response when the write timeout of the
HTTP server is just about to expire. This is surprisingly difficult to get right, since is
must be ensured that all output is fully flushed in time, which needs support from
multiple levels of the RPC handler stack:

The timeout response can't use chunked transfer-encoding because there is no way to write
the final terminating chunk. net/http writes it when the topmost handler returns, but the
timeout will already be over by the time that happens. We decided to disable chunked
encoding by setting content-length explicitly.

Gzip compression must also be disabled for timeout responses because we don't know the
true content-length before compressing all output, i.e. compression would reintroduce
chunked transfer-encoding.

* eth/filters, eth/tracers: add request cancellation checks (ethereum#26320)

This ensures that RPC method handlers will react to a timeout or
cancelled request soon after the event occurs.

Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>

* rpc: add limit for batch request items and response size (ethereum#26681)

This PR adds server-side limits for JSON-RPC batch requests. Before this change, batches
were limited only by processing time. The server would pick calls from the batch and
answer them until the response timeout occurred, then stop processing the remaining batch
items.

Here, we are adding two additional limits which can be configured:

- the 'item limit': batches can have at most N items
- the 'response size limit': batches can contain at most X response bytes

These limits are optional in package rpc. In Geth, we set a default limit of 1000 items
and 25MB response size.

When a batch goes over the limit, an error response is returned to the client. However,
doing this correctly isn't always possible. In JSON-RPC, only method calls with a valid
`id` can be responded to. Since batches may also contain non-call messages or
notifications, the best effort thing we can do to report an error with the batch itself is
reporting the limit violation as an error for the first method call in the batch. If a batch is
too large, but contains only notifications and responses, the error will be reported with
a null `id`.

The RPC client was also changed so it can deal with errors resulting from too large
batches. An older client connected to the server code in this PR could get stuck
until the request timeout occurred when the batch is too large. **Upgrading to a version
of the RPC client containing this change is strongly recommended to avoid timeout issues.**

For some weird reason, when writing the original client implementation, @fjl worked off of
the assumption that responses could be distributed across batches arbitrarily. So for a
batch request containing requests `[A B C]`, the server could respond with `[A B C]` but
also with `[A B] [C]` or even `[A] [B] [C]` and it wouldn't make a difference to the
client.

So in the implementation of BatchCallContext, the client waited for all requests in the
batch individually. If the server didn't respond to some of the requests in the batch, the
client would eventually just time out (if a context was used).

With the addition of batch limits into the server, we anticipate that people will hit this
kind of error way more often. To handle this properly, the client now waits for a single
response batch and expects it to contain all responses to the requests.

---------

Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>

* format

* ethereum, ethclient: add FeeHistory support (ethereum#25403)

Co-authored-by: Felix Lange <fjl@twurst.com>

* internal/ethapi: return error when requesting invalid trie key (ethereum#25762)

This change makes eth_getProof and eth_getStorageAt return an error when
the argument contains invalid hex in storage keys.

Co-authored-by: Felix Lange <fjl@twurst.com>

* internal/ethapi: handle odd length hex in decodeHash (ethereum#25883)

This change adds zero-padding (prefix) of odd nibbles in the decodeHash function. 

Co-authored-by: ty <ty@oncoder.com>

* eth/filters, ethclient/gethclient: add fullTx option to pending tx filter (ethereum#25186)

This PR adds a way to subscribe to the _full_ pending transactions, as opposed to just being notified about hashes.

In use cases where client subscribes to newPendingTransactions and gets txhashes only to then request the actual transaction, the caller can now shortcut that flow and obtain the transactions directly.

Co-authored-by: Felix Lange <fjl@twurst.com>

* graphql: check header first in blocks query (ethereum#24190)

Fixes ethereum#24167

New behaviour is that the endpoint returns results only for available
blocks without returning an error when it doesn't find a block. Note we
skip any block after a non-existent block.

This adds a header fetch for every block in range (even if header
is not needed). Alternatively, we could do the check in every field's
resolver method to avoid this overhead.

* graphql: embed *Resolver instead of backend interface (ethereum#25468)

This creates some infrastructure to share resources between graphql
API objects.

* eth/filters: fix getLogs for pending block (ethereum#24949)

* eth/filters: fix pending for getLogs

* add pending method to test backend

* fix block range validation

* eth/filters: add global block logs cache (ethereum#25459)

This adds a cache for block logs which is shared by all filters. The cache
size of is configurable using the `--cache.blocklogs` flag.

Co-authored-by: Felix Lange <fjl@twurst.com>

* eth/filters: send rpctransactions in pending-subscription (ethereum#26126)

This PR changes the pending tx subscription to return RPCTransaction types instead of normal Transaction objects. This will fix the inconsistencies with other tx returning API methods (i.e. getTransactionByHash), and also fill in the sender value for the tx.

co-authored by @s1na

* rpc: fix unmarshaling of null result in CallContext (ethereum#26723)

The change fixes unmarshaling of JSON null results into json.RawMessage.

---------

Co-authored-by: Jason Yuan <jason.yuan@curvegrid.com>
Co-authored-by: Jason Yuan <jason.yuan869@gmail.com>

* eth/filters: fix a breaking change and return rpctransaction (ethereum#26757)

* eth/filters: fix a breaking change and return rpctransaction

* eth/filters: fix test cases

---------

Co-authored-by: Catror <me@catror.com>

---------

Co-authored-by: Nicholas <nicholas.zhaoyu@gmail.com>
Co-authored-by: Nicholas Zhao <nicholas.zhao@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: Ahmet Avci <ahmetabdullahavci07@gmail.com>
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
Co-authored-by: Sina Mahmoodi <itz.s1na@gmail.com>
Co-authored-by: mmsqe <mavis@crypto.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: lightclient <14004106+lightclient@users.noreply.github.com>
Co-authored-by: TY <45994721+tylerK1294@users.noreply.github.com>
Co-authored-by: ty <ty@oncoder.com>
Co-authored-by: lmittmann <lmittmann@users.noreply.github.com>
Co-authored-by: Jason Yuan <jason.yuan@curvegrid.com>
Co-authored-by: Jason Yuan <jason.yuan869@gmail.com>
Co-authored-by: Yier <90763233+yierx@users.noreply.github.com>
Co-authored-by: Catror <me@catror.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Apparent Performance regression in eth_getLogs
6 participants