Discrepancy between block v3 claimed rewards and rewards POST endpoint

[michaelsproul]

Description

The latest version of blockdreamer using the v3 API shows a discrepancy between the rewards claimed by the v3 API and the rewards returned from POST /lighthouse/analysis/block_rewards. I expected they would be equal in almost all cases (i.e. whenever there are no slashings).

Opening this issue so we don't forget to investigate.

slot 8004800: block from lighthouse-subscribe-none with 75 attestations & purported reward 41559183 gwei
slot 8004799: canonical <=> lighthouse-subscribe-none distance: 7056
slot 8004799: canonical block is likely Lighthouse@7056 (two closest match)
rewards from lighthouse-subscribe-none: 36544081 gwei (att: 35078467 gwei)
most profitable block from ["lighthouse-subscribe-none"]
slot 8004801: block from lighthouse-subscribe-none with 70 attestations & purported reward 41852187 gwei
slot 8004800: canonical <=> lighthouse-subscribe-none distance: 11138
slot 8004800: canonical block is likely Lighthouse@11138 (two closest match)
rewards from lighthouse-subscribe-none: 36796747 gwei (att: 35360563 gwei)

This is a bit hard to read but it says:

• v3 API said the reward for slot 800 was 41559183
• the POST block reward API said the reward for slot 800 was 36796747 (~88.5%)

[0xalex88]

Just found out this too when manually checking some validator performance:

• validator 1073492
• epoch 254430

rewards api shows:

{
  "validator_index": "1073492",
  "head": "0",
  "target": "-4888",
  "source": "2619",
  "inactivity": "0"
}

and an ideal reward of

{
  "effective_balance": "32000000000",
  "head": "2525",
  "target": "4721",
  "source": "2619",
  "inactivity": "0"
}

but the balance increased by 9983 between epoch 254430 and 254431 and the attestation matched the head

[michaelsproul]

@0xalex88 this issue is about block rewards rather than attestation rewards, so it's a different bug (if a bug at all)

I think actually it isn't a bug, because that validator did fail to match head in epoch 254430. They attested to the block from 2 slots prior (https://beaconcha.in/slot/20433f2c22faa6707f104f44ce227f6cb6e51e029073946ce06483f5f7827d34), i.e. attesting to slot 8141758 at slot 8141760.

The reason the balance change from epoch 30 -> 31 doesn't show the penalty is that the rewards for attestations in epoch N are applied at the transition from N + 1 to N + 2 (because attestations for the previous epoch can be included for all of epoch N + 1). So we see the balance diff between 31 and 32:

curl -s "http://localhost:5052/eth/v1/beacon/states/8141792/validator_balances?id=1073492" | jq

{
  "execution_optimistic": false,
  "finalized": true,
  "data": [
    {
      "index": "1073492",
      "balance": "32014336347"
    }
  ]
}

curl -s "http://localhost:5052/eth/v1/beacon/states/8141824/validator_balances?id=1073492" | jq

{
  "execution_optimistic": false,
  "finalized": true,
  "data": [
    {
      "index": "1073492",
      "balance": "32014334078"
    }
  ]
}

The difference is 32014334078 - 32014336347 = -2269. This matches the rewards API: -4888 + 2619 = -2269.

[0xalex88]

@0xalex88 this issue is about block rewards rather than attestation rewards, so it's a different bug (if a bug at all)

I think actually it isn't a bug, because that validator did fail to match head in epoch 254430. They attested to the block from 2 slots prior (https://beaconcha.in/slot/20433f2c22faa6707f104f44ce227f6cb6e51e029073946ce06483f5f7827d34), i.e. attesting to slot 8141758 at slot 8141760.

you're right, my bad, I got confused looking at the 0 inclusion distance on beaconcha.in which is different than the difference between the attested block and when it's sent/included

[michaelsproul]

I found and mostly fixed the issue from the OP here: #5047. There is some subtlety around getting the results to match exactly which may not be worth it.

We also need to determine the impact of this change on blockprint which is the main consumer. I don't think it should make any (negative) difference as blockprint doesn't rely on the absolute reward values and their relative ordering should remain the same.