Added latency_percentiles_usec summary (A summary of latency percentile distribution per command) #652

filipecosta90 · 2022-05-25T22:18:59Z

As of Redis 7 ( specifically this PR redis/redis#9462 ) it possible exporting the per-command percentile distribution via the INFO LATENCYSTATS subcommand or INFO ALL. ( percentile distribution is not mergeable between cluster nodes )

Here's a sample of INFO ALL ( focusing on commandstats and latencystats ):

(...)
# Commandstats
cmdstat_rpop:calls=100000,usec=109759,usec_per_call=1.10,rejected_calls=0,failed_calls=0
cmdstat_set:calls=100000,usec=76770,usec_per_call=0.77,rejected_calls=0,failed_calls=0
cmdstat_zadd:calls=100000,usec=146737,usec_per_call=1.47,rejected_calls=0,failed_calls=0
cmdstat_mset:calls=100000,usec=381537,usec_per_call=3.82,rejected_calls=0,failed_calls=0
cmdstat_lpop:calls=100000,usec=130932,usec_per_call=1.31,rejected_calls=0,failed_calls=0
cmdstat_rpush:calls=100000,usec=69972,usec_per_call=0.70,rejected_calls=0,failed_calls=0
cmdstat_get:calls=100000,usec=69280,usec_per_call=0.69,rejected_calls=0,failed_calls=0
cmdstat_zpopmin:calls=100000,usec=49284,usec_per_call=0.49,rejected_calls=0,failed_calls=0
cmdstat_ping:calls=200000,usec=39815,usec_per_call=0.20,rejected_calls=0,failed_calls=0
cmdstat_spop:calls=100000,usec=48228,usec_per_call=0.48,rejected_calls=0,failed_calls=0
cmdstat_incr:calls=100000,usec=62637,usec_per_call=0.63,rejected_calls=0,failed_calls=0
cmdstat_lrange:calls=400000,usec=19436744,usec_per_call=48.59,rejected_calls=0,failed_calls=0
cmdstat_hset:calls=100000,usec=120657,usec_per_call=1.21,rejected_calls=0,failed_calls=0
cmdstat_config|get:calls=2,usec=26,usec_per_call=13.00,rejected_calls=0,failed_calls=0
cmdstat_lpush:calls=200000,usec=229105,usec_per_call=1.15,rejected_calls=0,failed_calls=0
cmdstat_sadd:calls=100000,usec=54369,usec_per_call=0.54,rejected_calls=0,failed_calls=0

# Errorstats

# Latencystats
latency_percentiles_usec_rpop:p50=1.003,p99=3.007,p99.9=10.047
latency_percentiles_usec_set:p50=1.003,p99=2.007,p99.9=3.007
latency_percentiles_usec_zadd:p50=1.003,p99=6.015,p99.9=17.023
latency_percentiles_usec_mset:p50=3.007,p99=8.031,p99.9=19.071
latency_percentiles_usec_lpop:p50=1.003,p99=4.015,p99.9=12.031
latency_percentiles_usec_rpush:p50=1.003,p99=3.007,p99.9=12.031
latency_percentiles_usec_get:p50=1.003,p99=2.007,p99.9=3.007
latency_percentiles_usec_zpopmin:p50=0.001,p99=2.007,p99.9=6.015
latency_percentiles_usec_ping:p50=0.001,p99=1.003,p99.9=2.007
latency_percentiles_usec_spop:p50=0.001,p99=2.007,p99.9=6.015
latency_percentiles_usec_incr:p50=1.003,p99=2.007,p99.9=3.007
latency_percentiles_usec_lrange:p50=53.247,p99=134.143,p99.9=154.623
latency_percentiles_usec_hset:p50=1.003,p99=5.023,p99.9=14.015
latency_percentiles_usec_config|get:p50=5.023,p99=21.119,p99.9=21.119
latency_percentiles_usec_lpush:p50=1.003,p99=5.023,p99.9=19.071
latency_percentiles_usec_sadd:p50=1.003,p99=2.007,p99.9=5.023

With the above info we can generate Prometheus compliant summaries ( one per command ).

# HELP redis_latency_percentiles_usec A summary of latency percentile distribution per command
# TYPE redis_latency_percentiles_usec summary
redis_latency_percentiles_usec{cmd="client|setname",quantile="50"} 5.023
redis_latency_percentiles_usec{cmd="client|setname",quantile="99"} 5.023
redis_latency_percentiles_usec{cmd="client|setname",quantile="99.9"} 5.023
redis_latency_percentiles_usec_sum{cmd="client|setname"} 5
redis_latency_percentiles_usec_count{cmd="client|setname"} 1
redis_latency_percentiles_usec{cmd="config|get",quantile="50"} 21.119
redis_latency_percentiles_usec{cmd="config|get",quantile="99"} 440.319
redis_latency_percentiles_usec{cmd="config|get",quantile="99.9"} 440.319
redis_latency_percentiles_usec_sum{cmd="config|get"} 465
redis_latency_percentiles_usec_count{cmd="config|get"} 3
redis_latency_percentiles_usec{cmd="get",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="get",quantile="99"} 2.007
redis_latency_percentiles_usec{cmd="get",quantile="99.9"} 3.007
redis_latency_percentiles_usec_sum{cmd="get"} 69280
redis_latency_percentiles_usec_count{cmd="get"} 100000
redis_latency_percentiles_usec{cmd="hset",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="hset",quantile="99"} 5.023
redis_latency_percentiles_usec{cmd="hset",quantile="99.9"} 14.015
redis_latency_percentiles_usec_sum{cmd="hset"} 120657
redis_latency_percentiles_usec_count{cmd="hset"} 100000
redis_latency_percentiles_usec{cmd="incr",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="incr",quantile="99"} 2.007
redis_latency_percentiles_usec{cmd="incr",quantile="99.9"} 3.007
redis_latency_percentiles_usec_sum{cmd="incr"} 62637
redis_latency_percentiles_usec_count{cmd="incr"} 100000
redis_latency_percentiles_usec{cmd="info",quantile="50"} 411.647
redis_latency_percentiles_usec{cmd="info",quantile="99"} 411.647
redis_latency_percentiles_usec{cmd="info",quantile="99.9"} 411.647
redis_latency_percentiles_usec_sum{cmd="info"} 411
redis_latency_percentiles_usec_count{cmd="info"} 1
redis_latency_percentiles_usec{cmd="lpop",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="lpop",quantile="99"} 4.015
redis_latency_percentiles_usec{cmd="lpop",quantile="99.9"} 12.031
redis_latency_percentiles_usec_sum{cmd="lpop"} 130932
redis_latency_percentiles_usec_count{cmd="lpop"} 100000
redis_latency_percentiles_usec{cmd="lpush",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="lpush",quantile="99"} 5.023
redis_latency_percentiles_usec{cmd="lpush",quantile="99.9"} 19.071
redis_latency_percentiles_usec_sum{cmd="lpush"} 229105
redis_latency_percentiles_usec_count{cmd="lpush"} 200000
redis_latency_percentiles_usec{cmd="lrange",quantile="50"} 53.247
redis_latency_percentiles_usec{cmd="lrange",quantile="99"} 134.143
redis_latency_percentiles_usec{cmd="lrange",quantile="99.9"} 154.623
redis_latency_percentiles_usec_sum{cmd="lrange"} 1.9436744e+07
redis_latency_percentiles_usec_count{cmd="lrange"} 400000
redis_latency_percentiles_usec{cmd="mset",quantile="50"} 3.007
redis_latency_percentiles_usec{cmd="mset",quantile="99"} 8.031
redis_latency_percentiles_usec{cmd="mset",quantile="99.9"} 19.071
redis_latency_percentiles_usec_sum{cmd="mset"} 381537
redis_latency_percentiles_usec_count{cmd="mset"} 100000
redis_latency_percentiles_usec{cmd="ping",quantile="50"} 0.001
redis_latency_percentiles_usec{cmd="ping",quantile="99"} 1.003
redis_latency_percentiles_usec{cmd="ping",quantile="99.9"} 2.007
redis_latency_percentiles_usec_sum{cmd="ping"} 39815
redis_latency_percentiles_usec_count{cmd="ping"} 200000
redis_latency_percentiles_usec{cmd="rpop",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="rpop",quantile="99"} 3.007
redis_latency_percentiles_usec{cmd="rpop",quantile="99.9"} 10.047
redis_latency_percentiles_usec_sum{cmd="rpop"} 109759
redis_latency_percentiles_usec_count{cmd="rpop"} 100000
redis_latency_percentiles_usec{cmd="rpush",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="rpush",quantile="99"} 3.007
redis_latency_percentiles_usec{cmd="rpush",quantile="99.9"} 12.031
redis_latency_percentiles_usec_sum{cmd="rpush"} 69972
redis_latency_percentiles_usec_count{cmd="rpush"} 100000
redis_latency_percentiles_usec{cmd="sadd",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="sadd",quantile="99"} 2.007
redis_latency_percentiles_usec{cmd="sadd",quantile="99.9"} 5.023
redis_latency_percentiles_usec_sum{cmd="sadd"} 54369
redis_latency_percentiles_usec_count{cmd="sadd"} 100000
redis_latency_percentiles_usec{cmd="set",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="set",quantile="99"} 2.007
redis_latency_percentiles_usec{cmd="set",quantile="99.9"} 3.007
redis_latency_percentiles_usec_sum{cmd="set"} 76770
redis_latency_percentiles_usec_count{cmd="set"} 100000
redis_latency_percentiles_usec{cmd="spop",quantile="50"} 0.001
redis_latency_percentiles_usec{cmd="spop",quantile="99"} 2.007
redis_latency_percentiles_usec{cmd="spop",quantile="99.9"} 6.015
redis_latency_percentiles_usec_sum{cmd="spop"} 48228
redis_latency_percentiles_usec_count{cmd="spop"} 100000
redis_latency_percentiles_usec{cmd="zadd",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="zadd",quantile="99"} 6.015
redis_latency_percentiles_usec{cmd="zadd",quantile="99.9"} 17.023
redis_latency_percentiles_usec_sum{cmd="zadd"} 146737
redis_latency_percentiles_usec_count{cmd="zadd"} 100000
redis_latency_percentiles_usec{cmd="zpopmin",quantile="50"} 0.001
redis_latency_percentiles_usec{cmd="zpopmin",quantile="99"} 2.007
redis_latency_percentiles_usec{cmd="zpopmin",quantile="99.9"} 6.015
redis_latency_percentiles_usec_sum{cmd="zpopmin"} 49284
redis_latency_percentiles_usec_count{cmd="zpopmin"} 100000

Note: controlling the exported percentiles on Redis

Notice that the exported percentiles on redis can be configured and that by default Redis exports the p50, p99 and p999. The logic behind this PR is agnostic to the exported percentiles.
As an example let's extend the exported percentiles in Redis to:

$ redis-cli 
127.0.0.1:6379> CONFIG SET latency-tracking-info-percentiles "0.0 50.0 75.0 95.0 99.0 100.0"
OK

The resulting exported summaries would be:

# HELP redis_latency_percentiles_usec A summary of latency percentile distribution per command
# TYPE redis_latency_percentiles_usec summary
redis_latency_percentiles_usec{cmd="client|setname",quantile="0"} 0
redis_latency_percentiles_usec{cmd="client|setname",quantile="50"} 3.007
redis_latency_percentiles_usec{cmd="client|setname",quantile="75"} 5.023
redis_latency_percentiles_usec{cmd="client|setname",quantile="95"} 5.023
redis_latency_percentiles_usec{cmd="client|setname",quantile="99"} 5.023
redis_latency_percentiles_usec{cmd="client|setname",quantile="100"} 5.023
redis_latency_percentiles_usec_sum{cmd="client|setname"} 8
redis_latency_percentiles_usec_count{cmd="client|setname"} 2
redis_latency_percentiles_usec{cmd="command",quantile="0"} 0
redis_latency_percentiles_usec{cmd="command",quantile="50"} 1654.783
redis_latency_percentiles_usec{cmd="command",quantile="75"} 1654.783
redis_latency_percentiles_usec{cmd="command",quantile="95"} 1654.783
redis_latency_percentiles_usec{cmd="command",quantile="99"} 1654.783
redis_latency_percentiles_usec{cmd="command",quantile="100"} 1654.783
redis_latency_percentiles_usec_sum{cmd="command"} 1651
redis_latency_percentiles_usec_count{cmd="command"} 1
redis_latency_percentiles_usec{cmd="config|get",quantile="0"} 0
redis_latency_percentiles_usec{cmd="config|get",quantile="50"} 21.119
redis_latency_percentiles_usec{cmd="config|get",quantile="75"} 360.447
redis_latency_percentiles_usec{cmd="config|get",quantile="95"} 440.319
redis_latency_percentiles_usec{cmd="config|get",quantile="99"} 440.319
redis_latency_percentiles_usec{cmd="config|get",quantile="100"} 440.319
redis_latency_percentiles_usec_sum{cmd="config|get"} 824
redis_latency_percentiles_usec_count{cmd="config|get"} 4
redis_latency_percentiles_usec{cmd="config|set",quantile="0"} 0
redis_latency_percentiles_usec{cmd="config|set",quantile="50"} 30.079
redis_latency_percentiles_usec{cmd="config|set",quantile="75"} 30.079
redis_latency_percentiles_usec{cmd="config|set",quantile="95"} 30.079
redis_latency_percentiles_usec{cmd="config|set",quantile="99"} 30.079
redis_latency_percentiles_usec{cmd="config|set",quantile="100"} 30.079
redis_latency_percentiles_usec_sum{cmd="config|set"} 30
redis_latency_percentiles_usec_count{cmd="config|set"} 1
redis_latency_percentiles_usec{cmd="get",quantile="0"} 0
redis_latency_percentiles_usec{cmd="get",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="get",quantile="75"} 1.003
redis_latency_percentiles_usec{cmd="get",quantile="95"} 1.003
redis_latency_percentiles_usec{cmd="get",quantile="99"} 2.007
redis_latency_percentiles_usec{cmd="get",quantile="100"} 8585.215
redis_latency_percentiles_usec_sum{cmd="get"} 69280
redis_latency_percentiles_usec_count{cmd="get"} 100000
redis_latency_percentiles_usec{cmd="hset",quantile="0"} 0
redis_latency_percentiles_usec{cmd="hset",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="hset",quantile="75"} 1.003
redis_latency_percentiles_usec{cmd="hset",quantile="95"} 3.007
redis_latency_percentiles_usec{cmd="hset",quantile="99"} 5.023
redis_latency_percentiles_usec{cmd="hset",quantile="100"} 1228.799
redis_latency_percentiles_usec_sum{cmd="hset"} 120657
redis_latency_percentiles_usec_count{cmd="hset"} 100000
redis_latency_percentiles_usec{cmd="incr",quantile="0"} 0
redis_latency_percentiles_usec{cmd="incr",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="incr",quantile="75"} 1.003
redis_latency_percentiles_usec{cmd="incr",quantile="95"} 1.003
redis_latency_percentiles_usec{cmd="incr",quantile="99"} 2.007
redis_latency_percentiles_usec{cmd="incr",quantile="100"} 74.239
redis_latency_percentiles_usec_sum{cmd="incr"} 62637
redis_latency_percentiles_usec_count{cmd="incr"} 100000
redis_latency_percentiles_usec{cmd="info",quantile="0"} 0
redis_latency_percentiles_usec{cmd="info",quantile="50"} 411.647
redis_latency_percentiles_usec{cmd="info",quantile="75"} 999.423
redis_latency_percentiles_usec{cmd="info",quantile="95"} 999.423
redis_latency_percentiles_usec{cmd="info",quantile="99"} 999.423
redis_latency_percentiles_usec{cmd="info",quantile="100"} 999.423
redis_latency_percentiles_usec_sum{cmd="info"} 1410
redis_latency_percentiles_usec_count{cmd="info"} 2
redis_latency_percentiles_usec{cmd="latency|latest",quantile="0"} 0
redis_latency_percentiles_usec{cmd="latency|latest",quantile="50"} 4.015
redis_latency_percentiles_usec{cmd="latency|latest",quantile="75"} 4.015
redis_latency_percentiles_usec{cmd="latency|latest",quantile="95"} 4.015
redis_latency_percentiles_usec{cmd="latency|latest",quantile="99"} 4.015
redis_latency_percentiles_usec{cmd="latency|latest",quantile="100"} 4.015
redis_latency_percentiles_usec_sum{cmd="latency|latest"} 4
redis_latency_percentiles_usec_count{cmd="latency|latest"} 1
redis_latency_percentiles_usec{cmd="lpop",quantile="0"} 0
redis_latency_percentiles_usec{cmd="lpop",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="lpop",quantile="75"} 2.007
redis_latency_percentiles_usec{cmd="lpop",quantile="95"} 3.007
redis_latency_percentiles_usec{cmd="lpop",quantile="99"} 4.015
redis_latency_percentiles_usec{cmd="lpop",quantile="100"} 54.015
redis_latency_percentiles_usec_sum{cmd="lpop"} 130932
redis_latency_percentiles_usec_count{cmd="lpop"} 100000
redis_latency_percentiles_usec{cmd="lpush",quantile="0"} 0
redis_latency_percentiles_usec{cmd="lpush",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="lpush",quantile="75"} 1.003
redis_latency_percentiles_usec{cmd="lpush",quantile="95"} 2.007
redis_latency_percentiles_usec{cmd="lpush",quantile="99"} 5.023
redis_latency_percentiles_usec{cmd="lpush",quantile="100"} 125.439
redis_latency_percentiles_usec_sum{cmd="lpush"} 229105
redis_latency_percentiles_usec_count{cmd="lpush"} 200000
redis_latency_percentiles_usec{cmd="lrange",quantile="0"} 0
redis_latency_percentiles_usec{cmd="lrange",quantile="50"} 53.247
redis_latency_percentiles_usec{cmd="lrange",quantile="75"} 64.255
redis_latency_percentiles_usec{cmd="lrange",quantile="95"} 112.127
redis_latency_percentiles_usec{cmd="lrange",quantile="99"} 134.143
redis_latency_percentiles_usec{cmd="lrange",quantile="100"} 2736.127
redis_latency_percentiles_usec_sum{cmd="lrange"} 1.9436744e+07
redis_latency_percentiles_usec_count{cmd="lrange"} 400000
redis_latency_percentiles_usec{cmd="mset",quantile="0"} 0
redis_latency_percentiles_usec{cmd="mset",quantile="50"} 3.007
redis_latency_percentiles_usec{cmd="mset",quantile="75"} 5.023
redis_latency_percentiles_usec{cmd="mset",quantile="95"} 6.015
redis_latency_percentiles_usec{cmd="mset",quantile="99"} 8.031
redis_latency_percentiles_usec{cmd="mset",quantile="100"} 139.263
redis_latency_percentiles_usec_sum{cmd="mset"} 381537
redis_latency_percentiles_usec_count{cmd="mset"} 100000
redis_latency_percentiles_usec{cmd="ping",quantile="0"} 0
redis_latency_percentiles_usec{cmd="ping",quantile="50"} 0.001
redis_latency_percentiles_usec{cmd="ping",quantile="75"} 0.001
redis_latency_percentiles_usec{cmd="ping",quantile="95"} 1.003
redis_latency_percentiles_usec{cmd="ping",quantile="99"} 1.003
redis_latency_percentiles_usec{cmd="ping",quantile="100"} 49.151
redis_latency_percentiles_usec_sum{cmd="ping"} 39815
redis_latency_percentiles_usec_count{cmd="ping"} 200000
redis_latency_percentiles_usec{cmd="rpop",quantile="0"} 0
redis_latency_percentiles_usec{cmd="rpop",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="rpop",quantile="75"} 1.003
redis_latency_percentiles_usec{cmd="rpop",quantile="95"} 2.007
redis_latency_percentiles_usec{cmd="rpop",quantile="99"} 3.007
redis_latency_percentiles_usec{cmd="rpop",quantile="100"} 52.223
redis_latency_percentiles_usec_sum{cmd="rpop"} 109759
redis_latency_percentiles_usec_count{cmd="rpop"} 100000
redis_latency_percentiles_usec{cmd="rpush",quantile="0"} 0
redis_latency_percentiles_usec{cmd="rpush",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="rpush",quantile="75"} 1.003
redis_latency_percentiles_usec{cmd="rpush",quantile="95"} 1.003
redis_latency_percentiles_usec{cmd="rpush",quantile="99"} 3.007
redis_latency_percentiles_usec{cmd="rpush",quantile="100"} 61.183
redis_latency_percentiles_usec_sum{cmd="rpush"} 69972
redis_latency_percentiles_usec_count{cmd="rpush"} 100000
redis_latency_percentiles_usec{cmd="sadd",quantile="0"} 0
redis_latency_percentiles_usec{cmd="sadd",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="sadd",quantile="75"} 1.003
redis_latency_percentiles_usec{cmd="sadd",quantile="95"} 1.003
redis_latency_percentiles_usec{cmd="sadd",quantile="99"} 2.007
redis_latency_percentiles_usec{cmd="sadd",quantile="100"} 27.007
redis_latency_percentiles_usec_sum{cmd="sadd"} 54369
redis_latency_percentiles_usec_count{cmd="sadd"} 100000
redis_latency_percentiles_usec{cmd="set",quantile="0"} 0
redis_latency_percentiles_usec{cmd="set",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="set",quantile="75"} 1.003
redis_latency_percentiles_usec{cmd="set",quantile="95"} 1.003
redis_latency_percentiles_usec{cmd="set",quantile="99"} 2.007
redis_latency_percentiles_usec{cmd="set",quantile="100"} 166.911
redis_latency_percentiles_usec_sum{cmd="set"} 76770
redis_latency_percentiles_usec_count{cmd="set"} 100000
redis_latency_percentiles_usec{cmd="slowlog|get",quantile="0"} 0
redis_latency_percentiles_usec{cmd="slowlog|get",quantile="50"} 3.007
redis_latency_percentiles_usec{cmd="slowlog|get",quantile="75"} 3.007
redis_latency_percentiles_usec{cmd="slowlog|get",quantile="95"} 3.007
redis_latency_percentiles_usec{cmd="slowlog|get",quantile="99"} 3.007
redis_latency_percentiles_usec{cmd="slowlog|get",quantile="100"} 3.007
redis_latency_percentiles_usec_sum{cmd="slowlog|get"} 3
redis_latency_percentiles_usec_count{cmd="slowlog|get"} 1
redis_latency_percentiles_usec{cmd="slowlog|len",quantile="0"} 0
redis_latency_percentiles_usec{cmd="slowlog|len",quantile="50"} 2.007
redis_latency_percentiles_usec{cmd="slowlog|len",quantile="75"} 2.007
redis_latency_percentiles_usec{cmd="slowlog|len",quantile="95"} 2.007
redis_latency_percentiles_usec{cmd="slowlog|len",quantile="99"} 2.007
redis_latency_percentiles_usec{cmd="slowlog|len",quantile="100"} 2.007
redis_latency_percentiles_usec_sum{cmd="slowlog|len"} 2
redis_latency_percentiles_usec_count{cmd="slowlog|len"} 1
redis_latency_percentiles_usec{cmd="spop",quantile="0"} 0
redis_latency_percentiles_usec{cmd="spop",quantile="50"} 0.001
redis_latency_percentiles_usec{cmd="spop",quantile="75"} 1.003
redis_latency_percentiles_usec{cmd="spop",quantile="95"} 1.003
redis_latency_percentiles_usec{cmd="spop",quantile="99"} 2.007
redis_latency_percentiles_usec{cmd="spop",quantile="100"} 52.223
redis_latency_percentiles_usec_sum{cmd="spop"} 48228
redis_latency_percentiles_usec_count{cmd="spop"} 100000
redis_latency_percentiles_usec{cmd="zadd",quantile="0"} 0
redis_latency_percentiles_usec{cmd="zadd",quantile="50"} 1.003
redis_latency_percentiles_usec{cmd="zadd",quantile="75"} 2.007
redis_latency_percentiles_usec{cmd="zadd",quantile="95"} 3.007
redis_latency_percentiles_usec{cmd="zadd",quantile="99"} 6.015
redis_latency_percentiles_usec{cmd="zadd",quantile="100"} 823.295
redis_latency_percentiles_usec_sum{cmd="zadd"} 146737
redis_latency_percentiles_usec_count{cmd="zadd"} 100000
redis_latency_percentiles_usec{cmd="zpopmin",quantile="0"} 0
redis_latency_percentiles_usec{cmd="zpopmin",quantile="50"} 0.001
redis_latency_percentiles_usec{cmd="zpopmin",quantile="75"} 1.003
redis_latency_percentiles_usec{cmd="zpopmin",quantile="95"} 1.003
redis_latency_percentiles_usec{cmd="zpopmin",quantile="99"} 2.007
redis_latency_percentiles_usec{cmd="zpopmin",quantile="100"} 112.127
redis_latency_percentiles_usec_sum{cmd="zpopmin"} 49284
redis_latency_percentiles_usec_count{cmd="zpopmin"} 100000

It's important to emphasize that o Redis we don't store summaries --- we preserve the entire sketch of data --- meaning that if you want to extend the exported percentiles it's retroactive to the current command stats already present on Redis =)

…le distribution per command)

oliver006 · 2022-05-26T01:37:42Z

Thanks for the PR, this is great. I'll try to review it in the new few days.
One heads up, the build is failing: https://cloud.drone.io/oliver006/redis_exporter/1535

filipecosta90 · 2022-05-26T10:13:06Z

Thanks for the PR, this is great. I'll try to review it in the new few days. One heads up, the build is failing: https://cloud.drone.io/oliver006/redis_exporter/1535

thank you @oliver006 .
WRT to the failing tests, It seems that now it's not related to the PR itself.

=== RUN   TestExtractInfoMetricsSentinel

    sentinels_test.go:25: Couldn't connect to "redis://redis-sentinel:26379": &net.OpError{Op:"dial", Net:"tcp", Source:net.Addr(nil), Addr:net.Addr(nil), Err:(*net.DNSError)(0xc00029c480)}

--- FAIL: TestExtractInfoMetricsSentinel (0.01s)

can you confirm?

oliver006 · 2022-05-26T12:23:25Z

I restarted the tests, sometimes that container doesn't come up in time and now it's looking better.

There's a new error:

=== RUN   TestLatencyStats

    exporter_test.go:147: couldn't setup redis for uri redis://redis7:6384, err: dial tcp: lookup redis7 on 127.0.0.11:53: no such host 

time="2022-05-26T12:13:43Z" level=error msg="Couldn't connect to redis instance"

    info_test.go:135: didn't find redis_latency_percentiles_usec

    exporter_test.go:186: couldn't setup redis, err: dial tcp: lookup redis7 on 127.0.0.11:53: no such host 

--- FAIL: TestLatencyStats (0.05s)

You need to add redis7 container to the .drone.yml file as well - the docker-compose file is just for local development and not used for the automated tests.

filipecosta90 · 2022-05-26T13:35:44Z

I restarted the tests, sometimes that container doesn't come up in time and now it's looking better.

There's a new error:
=== RUN   TestLatencyStats

    exporter_test.go:147: couldn't setup redis for uri redis://redis7:6384, err: dial tcp: lookup redis7 on 127.0.0.11:53: no such host 

time="2022-05-26T12:13:43Z" level=error msg="Couldn't connect to redis instance"

    info_test.go:135: didn't find redis_latency_percentiles_usec

    exporter_test.go:186: couldn't setup redis, err: dial tcp: lookup redis7 on 127.0.0.11:53: no such host 

--- FAIL: TestLatencyStats (0.05s)
You need to add redis7 container to the .drone.yml file as well - the docker-compose file is just for local development and not used for the automated tests.

done. thank you for the hint @oliver006 !

oliver006

Really nice PR - nice work!
One question about the parsing/registering though. Let me know what you think.

exporter/info.go

oliver006

🚀

oliver006 · 2022-05-30T12:27:43Z

@filipecosta90 - the tests didn't run cause the .drone.yml file was malformed. I fixed that and now I see the new latency tests failing:
https://cloud.drone.io/oliver006/redis_exporter/1544/1/16
info_test.go:135: didn't find redis_latency_percentiles_usec

Could you have a look please?

filipecosta90 · 2022-05-30T12:32:41Z

checking it 👍 . thank you!

filipecosta90 · 2022-05-30T16:02:23Z

@oliver006 WRT to:
#652 (comment)

Addressed it in:

#653

Added latency_percentiles_usec summary (A summary of latency percenti…

9c61168

…le distribution per command)

Fixed lint issues

5f9c7c7

Updated drone.yml to include redis7

1230e41

oliver006 reviewed May 28, 2022

View reviewed changes

exporter/info.go Outdated Show resolved Hide resolved

filipecosta90 added 2 commits May 30, 2022 11:29

Producing command latency summaries after info processing

dbafd60

Moving logic to generate latency summaries to a specific function

15a4e0a

filipecosta90 requested a review from oliver006 May 30, 2022 11:40

oliver006 approved these changes May 30, 2022

View reviewed changes

oliver006 merged commit 123fca5 into oliver006:master May 30, 2022

filipecosta90 deleted the percentiles branch May 30, 2022 12:18

filipecosta90 mentioned this pull request May 30, 2022

fixed TestLatencyStats() wrong check #653

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added latency_percentiles_usec summary (A summary of latency percentile distribution per command) #652

Added latency_percentiles_usec summary (A summary of latency percentile distribution per command) #652

filipecosta90 commented May 25, 2022

oliver006 commented May 26, 2022

filipecosta90 commented May 26, 2022

oliver006 commented May 26, 2022

filipecosta90 commented May 26, 2022

oliver006 left a comment

oliver006 left a comment

oliver006 commented May 30, 2022

filipecosta90 commented May 30, 2022

filipecosta90 commented May 30, 2022

Added latency_percentiles_usec summary (A summary of latency percentile distribution per command) #652

Added latency_percentiles_usec summary (A summary of latency percentile distribution per command) #652

Conversation

filipecosta90 commented May 25, 2022

Note: controlling the exported percentiles on Redis

oliver006 commented May 26, 2022

filipecosta90 commented May 26, 2022

oliver006 commented May 26, 2022

filipecosta90 commented May 26, 2022

oliver006 left a comment

Choose a reason for hiding this comment

oliver006 left a comment

Choose a reason for hiding this comment

oliver006 commented May 30, 2022

filipecosta90 commented May 30, 2022

filipecosta90 commented May 30, 2022