Besu FCU timeout

[jframe]

Besu is timing out on the FCU. Restarting Besu is the only way to fix the error.

Seeing a lot of logs with

io.vertx.core.VertxException: Thread blocked
        at java.base/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:221)
        at java.base/java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1864)
        at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
        at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
        at java.base/java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1898)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2072)
        at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.ExecutionEngineJsonRpcMethod.response(ExecutionEngineJsonRpcMethod.java:115)
        at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)
        at org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:41)

And in Teku

2024-04-12 02:03:19.185 ERROR - Failed to update fork choice.
java.util.concurrent.CompletionException: tech.pegasys.teku.ethereum.executionclient.response.InvalidRemoteResponseException: Invalid remote response from the execution client: TimeoutException
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:649) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?]
at tech.pegasys.teku.infrastructure.async.SafeFuture.lambda$propagateResult$3(SafeFuture.java:146) ~[teku-infrastructure-async-24.3.1.jar:24.3.1]
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) ~[?:?]
Caused by: tech.pegasys.teku.ethereum.executionclient.response.InvalidRemoteResponseException: Invalid remote response from the execution client: TimeoutException
at tech.pegasys.teku.ethereum.executionclient.response.ResponseUnwrapper.unwrapResponseOrThrow(ResponseUnwrapper.java:39) ~[teku-ethereum-executionclient-24.3.1.jar:24.3.1]
at tech.pegasys.teku.ethereum.executionclient.response.ResponseUnwrapper.unwrapExecutionClientResponseOrThrow(ResponseUnwrapper.java:22) ~[teku-ethereum-executionclient-24.3.1.jar:24.3.1]
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) ~[?:?]
... 7 more
2024-04-12 02:03:19.192 ERROR - Unable to import block: Execution Client returned an error: tech.pegasys.teku.ethereum.executionclient.response.InvalidRemoteResponseException: Invalid remote response from the execution client: TimeoutException

Besu logs:
besu.txt

Teku logs:
teku.txt

[non-fungible-nelson]

@ahamlat if you have a moment

[ahamlat]

We don't have too much data here to know what is causing the issue, but at least we can improve Besu code, because it is not timing out, Besu threads are waiting forever, and Teku is timing out.
I will create a small PR for this case.