Low validator effectiveness

[chris13524]

Describe the bug

I'm trying to use reth in my Rocket Pool validator, but I'm very often getting missed head votes, >0 inclusion distance, or sometimes even missed attestations. Was somewhere around 80% effective.

I switched to Nethermind a few weeks ago and my validator effectiveness immediately became basically perfect and has remained so. I just switched back to reth v1.2.2 (I thought maybe this was my issue) but unfortunately the validator performance is still quite poor once again.

I would love to use reth (really just because it's Rust) in my validator. How can I help debug this?

An example slot is 11,286,668 which while had a 0-block inclusion distance had an incorrect head vote.

Steps to reproduce

Use reth & Nimbus in Rocket Pool node.

Node logs

https://gist.github.com/chris13524/40075bd5a7055e50e77cb0cefd5c3cab

Platform(s)

Linux (ARM), Linux (x86)

Container Type

Docker

What version/commit are you on?

reth Version: 1.2.2
Commit SHA: 2f4c509
Build Timestamp: 2025-03-05T10:23:51.381756351Z
Build Features: asm_keccak,jemalloc
Build Profile: maxperf

What database version are you on?

Got an error:

2025-03-18T02:00:16.617616Z  INFO Initialized tracing, debug log directory: /root/.cache/reth/logs/mainnet
Error: Datadir does not exist: "/root/.local/share/reth/mainnet"

Location:
    /project/crates/cli/commands/src/db/mod.rs:73:9

Which chain / network are you on?

mainnet

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

[prune]
block_interval = 5

[prune.segments]
sender_recovery = "full"

[prune.segments.receipts]
before = 11052984

[prune.segments.account_history]
distance = 10064

[prune.segments.storage_history]
distance = 10064

[prune.segments.receipts_log_filter.0x00000000219ab540356cbb839cbe05303d7705fa]
before = 11052984

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[jenpaff]

hey @chris13524 ! could you give more details so we can try and reproduce:

1. which version of lighthouse are you running? we received a report that with v7.0 aggressive pruning parameters are set which could lead to higher cpu usage High read IO due to blob pruning sigp/lighthouse#7100 it would be helpful if you could share your grafana dashboard, so we could identify if thats the root cause.
2. can you share some details on hardware spec where you are running your validator? This link is a good ref for good,bad,ugly server spec https://gist.github.com/yorickdowne/f3a3e79a573bf35767cd002cc977b038

[chris13524]

1. which version of lighthouse are you running?

Apologies, I'm using Nimbus as my consensus client. I meant to say for execution client I switched from Reth to Nethermind and the issue subsided. Have not tested Lighthouse.

I'm using a WD Black SN850X which is in the "good" category. Otherwise running on a 4-core NUC with 16 GB of RAM. Plenty of headroom. And again, hardware specs are not an issue for Nethermind

[jenpaff]

1. which version of lighthouse are you running?

Apologies, I'm using Nimbus as my consensus client. I meant to say for execution client I switched from Reth to Nethermind and the issue subsided. Have not tested Lighthouse.

I'm using a WD Black SN850X which is in the "good" category. Otherwise running on a 4-core NUC with 16 GB of RAM. Plenty of headroom. And again, hardware specs are not an issue for Nethermind

Ok thanks- we're looking into it. Would you be able to share any screenshots from your Grafana dashboard?

[jenpaff]

@chris13524 would you try bumping reth version to the latest and let us know if you see any improvements?

[chris13524]

@jenpaff I tested v1.3.4 for over a day and it doesn't look like much has changed. Getting 85% effectiveness and 67% efficiency. Plenty of missed or late attestations, and plenty of incorrect head votes.

[chris13524]

As requested here are some Grafana screenshots from the past 12 hours of reth operation. All seem nominal to me. Perhaps disk space is of concern but I have 140 GB free.

[jenpaff]

@chris13524 could you give it another go and update to the latest version. we've received good reports from stakers running Reth <> Nimbus https://t.me/paradigm_reth/40580