[Metrics] rpc_duration_xxxx Indirectly return zero data

[J1a-wei]

System information

Geth version: v1.10.26

Geth Args

exec geth --authrpc.jwtsecret=/secret/jwtsecret --authrpc.addr=0.0.0.0 --authrpc.port=8551 --authrpc.vhosts=* --http --http.addr=0.0.0.0 --http.port=8599 --http.corsdomain= --http.vhosts=* --http.api=web3,eth,net,engine --nat=extip:$(POD_IP) --port=30303 --ws --ws.addr=0.0.0.0 --ws.port=8546 --ws.origins=* --ws.api=web3,eth,net,engine --datadir=/data/ethereum --ethash.dagdir=/data/ethereum/.ethash --ipcdisable --goerli "--syncmode=snap" "--metrics" "--pprof" "--pprof.addr=0.0.0.0" "--pprof.port=6060" "--metrics.expensive"

• http.port=8599
• metrics port= 6060

I tried to call eth_blockNumber rpc many times. When requesting the metrics interface, the first time I have data, and later I access the metrics stats again as 0.

# curl eth_blockNumber jsonRPC 
curl -i -X POST  -H 'content-type: application/json' --data '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'  http://localhost:8599
HTTP/1.1 200 OK
Content-Type: application/json
Date: Thu, 29 Dec 2022 11:56:13 GMT
Content-Length: 45

{"jsonrpc":"2.0","id":1,"result":"0x7d70e8"}

# check metrics 
curl localhost:6060/debug/metrics/prometheus | grep eth_blockNumber
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 48729  100 48729    0     0  51151      0 --:--:-- --:--:-- --:--:-- 51347
# TYPE rpc_duration_eth_blockNumber_success_count counter
rpc_duration_eth_blockNumber_success_count 1
# TYPE rpc_duration_eth_blockNumber_success summary
rpc_duration_eth_blockNumber_success {quantile="0.5"} 27
rpc_duration_eth_blockNumber_success {quantile="0.75"} 27
rpc_duration_eth_blockNumber_success {quantile="0.95"} 27
rpc_duration_eth_blockNumber_success {quantile="0.99"} 27
rpc_duration_eth_blockNumber_success {quantile="0.999"} 27
rpc_duration_eth_blockNumber_success {quantile="0.9999"} 27

# check metrics again  
curl localhost:6060/debug/metrics/prometheus | grep eth_blockNumber
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 48747  100 48747    0     0  36100      0  0:00:01  0:00:01 --:--:-- 36189
# TYPE rpc_duration_eth_blockNumber_success_count counter
rpc_duration_eth_blockNumber_success_count 0
# TYPE rpc_duration_eth_blockNumber_success summary
rpc_duration_eth_blockNumber_success {quantile="0.5"} 0
rpc_duration_eth_blockNumber_success {quantile="0.75"} 0
rpc_duration_eth_blockNumber_success {quantile="0.95"} 0
rpc_duration_eth_blockNumber_success {quantile="0.99"} 0
rpc_duration_eth_blockNumber_success {quantile="0.999"} 0
rpc_duration_eth_blockNumber_success {quantile="0.9999"} 0


Even if I access the interface again, the query metrics are all 0

[holiman]

I cannot repro your error. Here's my attempt, where I invoke eth_blokcNumber and then fetch the metric. Fetching it zeroes the meter, and it becomes non-zero again when I invoke the method another time.

Geth started with geth --dev --pprof --pprof.addr=0.0.0.0 --pprof.port=6060 --metrics --http

[user@work ~]$ curl localhost:6060/debug/metrics/prometheus 2>/dev/null | grep eth_blockNumber | grep 0.5 
rpc_duration_eth_blockNumber_success {quantile="0.5"} 0
[user@work ~]$ curl http://localhost:8545/  -X POST -H "Content-Type: application/json" --data '{"method":"eth_blockNumber","params":[],"id":1,"jsonrpc":"2.0"}'
{"jsonrpc":"2.0","id":1,"result":"0x0"}
[user@work ~]$ curl localhost:6060/debug/metrics/prometheus 2>/dev/null | grep eth_blockNumber | grep 0.5 
rpc_duration_eth_blockNumber_success {quantile="0.5"} 94
[user@work ~]$ curl localhost:6060/debug/metrics/prometheus 2>/dev/null | grep eth_blockNumber | grep 0.5 
rpc_duration_eth_blockNumber_success {quantile="0.5"} 0
[user@work ~]$ curl http://localhost:8545/  -X POST -H "Content-Type: application/json" --data '{"method":"eth_blockNumber","params":[],"id":1,"jsonrpc":"2.0"}'
{"jsonrpc":"2.0","id":1,"result":"0x0"}
[user@work ~]$ curl localhost:6060/debug/metrics/prometheus 2>/dev/null | grep eth_blockNumber | grep 0.5 
rpc_duration_eth_blockNumber_success {quantile="0.5"} 36
[user@work ~]$ curl localhost:6060/debug/metrics/prometheus 2>/dev/null | grep eth_blockNumber | grep 0.5 
rpc_duration_eth_blockNumber_success {quantile="0.5"} 0
[user@work ~]$ curl http://localhost:8545/  -X POST -H "Content-Type: application/json" --data '{"method":"eth_blockNumber","params":[],"id":1,"jsonrpc":"2.0"}'
{"jsonrpc":"2.0","id":1,"result":"0x0"}
[user@work ~]$ curl localhost:6060/debug/metrics/prometheus 2>/dev/null | grep eth_blockNumber | grep 0.5 
rpc_duration_eth_blockNumber_success {quantile="0.5"} 51
[user@work ~]$ curl localhost:6060/debug/metrics/prometheus 2>/dev/null | grep eth_blockNumber | grep 0.5 
rpc_duration_eth_blockNumber_success {quantile="0.5"} 0
[user@work ~]$ curl http://localhost:8545/  -X POST -H "Content-Type: application/json" --data '{"method":"eth_blockNumber","params":[],"id":1,"jsonrpc":"2.0"}'
{"jsonrpc":"2.0","id":1,"result":"0x0"}
[user@work ~]$ curl localhost:6060/debug/metrics/prometheus 2>/dev/null | grep eth_blockNumber | grep 0.5 
rpc_duration_eth_blockNumber_success {quantile="0.5"} 35
[user@work ~]$ curl localhost:6060/debug/metrics/prometheus 2>/dev/null | grep eth_blockNumber | grep 0.5 
rpc_duration_eth_blockNumber_success {quantile="0.5"} 0

[holiman]

As for "why does the histogram reset", see #26356

[holiman]

I first interpreted your report as you were saying that even if you again make a call to eth_blockNumber, it does not re-arm the meter.
Upon reading it again, however, I interpret it as you are reporting that the histogram resets when you fetch it.

This is a known issue, and will be resolved by the PR linked above.

[J1a-wei]

Thanks for answer, but the metrics type rpc is summary not histogram. So the summary indicator type will be reset as well? Also, it's not just this one rpc metrics, there are many but I didn't list them @holiman

[vehicles4real]

Can someone please help me with my wallet and my crypto I need it so 😔 badly right now.

…

On Thu, Dec 29, 2022, 9:38 PM Jiawei Zhang ***@***.***> wrote: ok, I see that, but the metrics type rpc is summary not histogram. So the summary indicator type will be reset as well? Also, it's not just this one rpc metrics, there are many but I didn't list them @holiman <https://github.com/holiman> — Reply to this email directly, view it on GitHub <#26393 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUKO374PRNL3SAZ6G7S62ZLWPZKLVANCNFSM6AAAAAATMBNJ2M> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[joshuacolvin0]

All the "rpc_duration_*" metrics are pulled from histogram data structures, and are all currently reset every time the metrics are scraped. The PR referenced above changes histogram behavior so that there is a proper sliding window for the histogram, instead of just resetting on each scrape.

@0xRook1e