debug_traceBlockByHash in parallel -> "Empty reply from server"

[vikstrous2]

System information

Geth version: v1.10.3
OS & Version: Linux
Commit hash : (if develop)

Expected behaviour

Geth gracefully traces the same block in parallel or blocks one of the trace requests and sequentially traces the same block.

Actual behaviour

Geth returns nothing (not even hearders) when tracing the same block in parallel

Steps to reproduce the behaviour

1. Run a mainnet node in gc archive mode.
2. Trace the same block twice in parallel from two terminals. Example command:

curl http://localhost:8545 -d '{"jsonrpc": "2.0", "id": "3", "method": "debug_traceBlockByHash", "params": ["0x34990bd2e00b5cba332051402675e4c19793851d7c26436751b1572bac951bf3", {"disableMemory":true, "disableStack":true, "disableStorage":true, "fullStorage":false, "timeout":"120s", "tracer": "callTracer"}]}' -H 'Content-type: application/json' -v | wc

* Expire in 0 ms for 6 (transfer 0x559b74672fb0)
* Expire in 1 ms for 1 (transfer 0x559b74672fb0)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Expire in 0 ms for 1 (transfer 0x559b74672fb0)
* Expire in 1 ms for 1 (transfer 0x559b74672fb0)
* Expire in 0 ms for 1 (transfer 0x559b74672fb0)
* Expire in 0 ms for 1 (transfer 0x559b74672fb0)
* Expire in 1 ms for 1 (transfer 0x559b74672fb0)
* Expire in 0 ms for 1 (transfer 0x559b74672fb0)
* Expire in 0 ms for 1 (transfer 0x559b74672fb0)
* Expire in 0 ms for 1 (transfer 0x559b74672fb0)
*   Trying ::1...
* TCP_NODELAY set
* Expire in 150000 ms for 3 (transfer 0x559b74672fb0)
* Expire in 200 ms for 4 (transfer 0x559b74672fb0)
* Connected to localhost (::1) port 8545 (#0)
> POST / HTTP/1.1
> Host: localhost:8545
> User-Agent: curl/7.64.0
> Accept: */*
> Content-type: application/json
> Content-Length: 278
>
} [278 bytes data]
* upload completely sent off: 278 out of 278 bytes
100   278    0     0  100   278      0      8  0:00:34  0:00:31  0:00:03     0* Empty reply from server
100   278    0     0  100   278      0      8  0:00:34  0:00:31  0:00:03     0
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
      0       0       0

Backtrace

n/a

[MariusVanDerWijden]

Hmm that's weird. I tried to reproduce but can't.
I ran the following in for different terminals for ~5 minutes and didn't see an empty reply even once.

for i in {1..10000}; do curl http://localhost:8546 -d '{"jsonrpc": "2.0", "id": "3", "method": "debug_traceBlockByNumber", "params": ["0x441845", {"disableMemory":true, "disableStack":true, "disableStorage":true, "fullStorage":false, "timeout":"120s", "tracer": "callTracer"}]}' -H 'Content-type: application/json' ; done 

I can't find the hash that you're looking for on etherscan either: https://etherscan.io/tx/0x34990bd2e00b5cba332051402675e4c19793851d7c26436751b1572bac951bf3
Are you sure that this is a valid blockhash?

[vikstrous2]

This is the block: https://etherscan.io/block/0x34990bd2e00b5cba332051402675e4c19793851d7c26436751b1572bac951bf3

I'm using debug_traceBlockByHash, not debug_traceBlockByNumber. The choice of block might matter.

[jayuuza]

I have a similar issue here #22869. I found that debug_traceBlockByHash fails on transactions that use large amounts of gas. Might be related

[vikstrous2]

@jayuuza yeah, that looks similar. We have seen empty replies from single requests, especially on ropsten.

BTW, while debugging this, I noticed many missing error checks in calls to codec.writeJSON/conn.writeJSON(). We are almost certainly triggering one of these cases. We spent some time debugging and this code path seems suspicious, but I don't know yet what's going on. We are running a custom geth build with extra debug info every time writeJSON fails. I'll report back if we find out what error code path is being triggered.

[holiman]

Duplicate of #21430

[s1na]

Fixed via #25457