Teku 24.8.0 VC & Teku 24.10.1 BN Incompatibility

[NeoPlays]

Description

I'm aware that running different version combination is not best practice, but i think this is still worth mentioning:
Running a Teku VC 24.8.0 with a Teku BN 24.10.1 (with docker) leads to failing block proposals:

2024-10-14 07:07:48.529 INFO - Received block for slot 10172137, block rewards 0.046197 ETH, execution payload value 0.239068 ETH
2024-10-14 07:07:48.537 INFO - Block for slot 10172137 was blinded and will only be sent to the beacon node which created it.
2024-10-14 07:07:48.545 ERROR - �[31mValidator *** Failed to produce block Slot: 10172137 Validator: 95802ae�[0m
java.lang.IllegalArgumentException: Invalid params response from Beacon Node API (url = http://<ip>:5051/eth/v2/beacon/blinded_blocks?broadcast_validation=gossip, status = 400, message = Missing required header value for (Eth-Consensus-Version))
at tech.pegasys.teku.validator.remote.typedef.ResponseHandler.defaultBadRequestHandler(ResponseHandler.java:161) ~[teku-validator-remote-24.8.0.jar:24.8.0]
at tech.pegasys.teku.validator.remote.typedef.ResponseHandler.handleResponse(ResponseHandler.java:125) ~[teku-validator-remote-24.8.0.jar:24.8.0]
at tech.pegasys.teku.validator.remote.typedef.handlers.AbstractTypeDefRequest.executeCall(AbstractTypeDefRequest.java:188) ~[teku-validator-remote-24.8.0.jar:24.8.0]
at tech.pegasys.teku.validator.remote.typedef.handlers.AbstractTypeDefRequest.postJson(AbstractTypeDefRequest.java:150) ~[teku-validator-remote-24.8.0.jar:24.8.0]
at tech.pegasys.teku.validator.remote.typedef.handlers.SendSignedBlockRequest.sendSignedBlockAsJson(SendSignedBlockRequest.java:121) ~[teku-validator-remote-24.8.0.jar:24.8.0]
at tech.pegasys.teku.validator.remote.typedef.handlers.SendSignedBlockRequest.sendSignedBlock(SendSignedBlockRequest.java:79) ~[teku-validator-remote-24.8.0.jar:24.8.0]
at tech.pegasys.teku.validator.remote.typedef.OkHttpValidatorTypeDefClient.sendSignedBlock(OkHttpValidatorTypeDefClient.java:141) ~[teku-validator-remote-24.8.0.jar:24.8.0]
at tech.pegasys.teku.validator.remote.RemoteValidatorApiHandler.lambda$sendSignedBlock$11(RemoteValidatorApiHandler.java:266) ~[teku-validator-remote-24.8.0.jar:24.8.0]
at tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:80) ~[teku-infrastructure-async-24.8.0.jar:24.8.0]
at tech.pegasys.teku.validator.remote.RemoteValidatorApiHandler.sendRequest(RemoteValidatorApiHandler.java:397) ~[teku-validator-remote-24.8.0.jar:24.8.0]
at tech.pegasys.teku.validator.remote.RemoteValidatorApiHandler.lambda$sendRequest$28(RemoteValidatorApiHandler.java:392) ~[teku-validator-remote-24.8.0.jar:24.8.0]
at tech.pegasys.teku.infrastructure.async.SafeFuture.of(SafeFuture.java:72) ~[teku-infrastructure-async-24.8.0.jar:24.8.0]
at tech.pegasys.teku.infrastructure.async.ScheduledExecutorAsyncRunner.lambda$createRunnableForAction$1(ScheduledExecutorAsyncRunner.java:124) ~[teku-infrastructure-async-24.8.0.jar:24.8.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]

With 500 Keys on this Teku VC instance, we missed 9 proposals over the last weekend.

[lucassaldanha]

I'm aware that running different version combinations is not best practice, but i think this is still worth mentioning

You are right! We do not recommend running different versions. All our testing is done on "parallel" versions so it is risky.

Thanks for bringing this to our attention. Will take a look to make sure it isn't related to a bug somewhere.

May I ask why you did not update your VC and only BN?

[NeoPlays]

May I ask why you did not update your VC and only BN?

So we're running Nodes on a bigger scale, therefore we're seperating VCs from Fullnodes. I was working on this Fullnode and decided to also run updates. As we're running fallbacks as well, i thought that if there was an issue that i couldn't spot immediately, the fallbacks would take care of it. But in this case it's a bit more complex than this.

Updating VCs with doppelgägner protection enabled also results in longer downtime.