Frequent "Attestation queue full" error.

[zheli]

Description

We started to see more frequent "attestation queue full" errors from our beacon nodes. I am a bit worried that it would affect our successful attestation rate.

Version

Lighthouse v3.5.1

Present Behaviour

More and more frequent "attestation queue full" errors are seen.

Expected Behaviour

Not seeing that errors at all.

Steps to resolve

Our beacon node is running in a data center with 8Ghz intel CPU and 8GB ram.

Screenshot

[michaelsproul]

I saw your comment on #4126. Are you running entirely web3signer validators, and making frequent calls to the validator client HTTP API?

We were thinking of scaling up the size of these queues independently, as it seems other users are filling them as well.

What's your set up for your execution node? Do you seem warnings or errors related to it? I'm guessing it's on a different machine?

[zheli]

Thanks for following up!

We are not using web3signer at the moment, we use vouch as validator and MEV builder. Not sure how often it calls the validator HTTP API, what log message should I search to find that out?

Our EL is geth. All clients are on their own servers. I haven't seen any errors in geth log when the attestation queue full error happened.

Sometimes, there are a few warnings or errors prior to attestation queue full message, but I am not sure if they are related. It might be request timeouts to EL, or beacon chain re-org.

[zheli]

I saw your comment on #4126. Are you running entirely web3signer validators, and making frequent calls to the validator client HTTP API?

We were thinking of scaling up the size of these queues independently, as it seems other users are filling them as well.

What's your set up for your execution node? Do you seem warnings or errors related to it? I'm guessing it's on a different machine?

Is the queue size fixed/hardcoded? Will we have a bigger queue if giving more memory to the machine?

[michaelsproul]

@zheli The queue size is fixed (hardcoded). You could try to prevent it from filling by allocating more/faster CPU cores. That way items will be cleared from the queue more quickly.

I recommend running the non-portable binary from our release page (or the :v4.0.2-rp.0-modern docker image) and using a CPU with SHA extensions. AMD tends to support SHA extensions better than Intel. If you're compiling yourself use the maxperf profile as described here: https://lighthouse-book.sigmaprime.io/installation-source.html#compilation-profiles

[michaelsproul]

Just to clarify, when you say all clients have their own machines, does that mean the Lighthouse BN and Geth are on different machines? What's the latency like between them (measured with ping is fine).

[michaelsproul]

Closing as stale. There's nothing we can really directly action based on this issue.

Possibly relevant:

• Improve beacon processor scheduling #6291