Skip to content

[Bug]: lost telemetry after upgrade to v0.50.7 #20992

Closed

Description

Is there an existing issue for this?

  • I have searched the existing issues

What happened?

After upgrading from v0.46.13 v0.50.7 we seem to have lost some cosmos layer telemetry.

Configuration

# config.toml
[instrumentation]
prometheus = true
prometheus_listen_addr = ":26660"
max_open_connections = 3
namespace = "cometbft"

# app.toml
[telemetry]
service-name = ""
enabled = true
enable-hostname = true
enable-hostname-label = true
enable-service-label = false
prometheus-retention-time = 60
global-labels = [
  [
    "chain_id",
    "pio-mainnet-1"
  ]
]

Metrics before upgrade

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 75520    0 75520    0     0   363k      0 --:--:-- --:--:-- --:--:--  365k
# HELP store_iavl_commit store_iavl_commit
# TYPE store_iavl_commit summary
store_iavl_commit{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.5"} 0.37117999792099
store_iavl_commit{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.9"} 0.6181100010871887
store_iavl_commit{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.99"} 2.5882089138031006
store_iavl_commit_sum{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1"} 258902.27967266366
store_iavl_commit_count{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1"} 677424
# HELP store_iavl_delete store_iavl_delete
# TYPE store_iavl_delete summary
store_iavl_delete{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.5"} 0.02745000086724758
store_iavl_delete{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.9"} 0.029260000213980675
store_iavl_delete{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.99"} 0.029260000213980675
store_iavl_delete_sum{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1"} 72579.197542749
store_iavl_delete_count{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1"} 76053
# HELP store_iavl_get store_iavl_get
# TYPE store_iavl_get summary
store_iavl_get{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.5"} 0.002409999957308173
store_iavl_get{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.9"} 0.0037499999161809683
store_iavl_get{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1",quantile="0.99"} 0.012059999629855156
store_iavl_get_sum{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1"} 2.6048887388131223e+06
store_iavl_get_count{chain_id="pio-mainnet-1",host="pio-mainnet-indexed-archived-fullblock-1"} 3.09507892e+08

Metrics after upgrade

curl http://localhost:26660/metrics  | rg iavl
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  103k    0  103k    0     0   533k      0 --:--:-- --:--:-- --:--:--  534k

Cosmos SDK Version

v0.50.7

How to reproduce?

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Labels

Type

No type

Projects

  • Status

    🥳 Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions