Prometheus not show any values in http://localhost:1234/metrics

[indramal]

This is my Pingora server code and I added Prometheus but I can not see any values using http://localhost:1234/metrics URL. How to solve this issue? I can see "my_counter" value increasing by println!("my_counter: {}", MY_COUNTER.get()); code. But nothing show in metrics.

I check Prometheus server run or not,

ss -tuln | grep 1234              
tcp   LISTEN 0      4096                            0.0.0.0:1234       0.0.0.0:* 

This is my code:

use async_trait::async_trait;
use clap::Parser;
use log::info;
use once_cell::sync::Lazy;
use pingora_core::Result;
use pingora_core::listeners::tls::TlsSettings;
use pingora_core::server::Server;
use pingora_core::server::configuration::Opt;
use pingora_core::services::background::background_service;
use pingora_core::services::listening::Service;
use pingora_core::upstreams::peer::HttpPeer;
use pingora_load_balancing::{LoadBalancer, health_check, selection::RoundRobin};
use pingora_proxy::http_proxy_service;
use pingora_proxy::{ProxyHttp, Session};
use prometheus::{IntCounter, register_int_counter};
use std::{sync::Arc, time::Duration};

pub struct LB(Arc<LoadBalancer<RoundRobin>>); // Load Balancer with Round Robin selection

static MY_COUNTER: Lazy<IntCounter> =
    Lazy::new(|| register_int_counter!("my_counter", "my counter").unwrap());

// Create Load Balance Proxy
#[async_trait]
impl ProxyHttp for LB {
    // For this small example, we don't need context storage
    type CTX = ();
    fn new_ctx(&self) -> Self::CTX {}

    // Upstream_peer returns the address where the request should be proxied to.
    async fn upstream_peer(&self, _session: &mut Session, _ctx: &mut ()) -> Result<Box<HttpPeer>> {
        let upstream = self
            .0
            .select(b"", 256) // hash doesn't matter
            .unwrap(); // use the select() method for the LoadBalancer to round-robin across the upstream IPs.

        info!("upstream peer is: {:?}", upstream);

        // Set SNI to one.one.one.one
        let peer = Box::new(HttpPeer::new(
            upstream,
            false,
            "one.one.one.one".to_string(),
        )); // Set SNI to one.one.one.one and disable TLS (HTTP) with flase / enable TLS (HTTPS) with true
        Ok(peer)
    }

    // In order for the 1.1.1.1 backends to accept our requests, a host header must be present.
    // Adding this header can be done by the upstream_request_filter() callback which modifies the
    // request header after the connection to the backends are established and before the request header is sent.
    async fn upstream_request_filter(
        &self,
        _session: &mut Session,
        upstream_request: &mut pingora_http::RequestHeader,
        _ctx: &mut Self::CTX,
    ) -> Result<()> {
        upstream_request
            .insert_header("Host", "one.one.one.one")
            .unwrap();
        Ok(())
    }

    async fn logging(
        &self,
        session: &mut Session,
        _e: Option<&pingora_core::Error>,
        ctx: &mut Self::CTX,
    ) {
        let response_code = session
            .response_written()
            .map_or(0, |resp| resp.status.as_u16());
        info!(
            "{} response code: {response_code}",
            self.request_summary(session, ctx)
        );

        MY_COUNTER.inc();
        println!("my_counter: {}", MY_COUNTER.get());
    }
}
// RUST_LOG=INFO cargo run --example load_balancer
fn main() {
    env_logger::init();

    MY_COUNTER.inc();
    println!("my_counter: {}", MY_COUNTER.get());

    let opt = Opt::parse_args(); // read command line arguments, so cargo run -- -h command "-- -h" to see help
    let mut my_server = Server::new(Some(opt)).unwrap(); // create server with command line arguments
    my_server.bootstrap();

    // All list of array upstreams servers to load balance
    let mut upstreams = LoadBalancer::try_from_iter(["127.0.0.1:3000"]).unwrap();

    // ===================
    // Health Check
    // ===================

    // We add health check in the background so that the bad server is never selected.
    let hc = health_check::TcpHealthCheck::new();
    upstreams.set_health_check(hc);
    upstreams.health_check_frequency = Some(Duration::from_secs(1));

    let background = background_service("health check", upstreams);

    let upstreams = background.task();

    // ===================
    // Load Balance Proxy
    // ===================

    let mut lb = http_proxy_service(&my_server.configuration, LB(upstreams));
    lb.add_tcp("0.0.0.0:6188");

    // ===================
    // TLS with H2 enabled to load balance proxy
    // ===================

    let cert_path = format!("{}/certs/cert.pem", env!("CARGO_MANIFEST_DIR"));
    let key_path = format!("{}/certs/key.pem", env!("CARGO_MANIFEST_DIR"));

    let mut tls_settings = TlsSettings::intermediate(&cert_path, &key_path).unwrap();
    tls_settings.enable_h2(); // This work HTTP/2.0 and HTTP/1.1 (Check it)
    lb.add_tls_with_settings("0.0.0.0:6189", None, tls_settings);

    // ===================
    // Prometheus HTTP Service
    // ===================

    let mut prometheus_service_http = Service::prometheus_http_service();
    prometheus_service_http.add_tcp("0.0.0.0:1234");

    // ===================
    // Run Server
    // ===================

    my_server.add_service(prometheus_service_http); // add prometheus http service to server
    my_server.add_service(lb); // add load balance proxy to server
    my_server.add_service(background); // add health check to server
    my_server.run_forever(); // spawn all the runtime threads and block the main thread until the server is ready to exit.
}

This is my Toml file:

[package]
name = "test_pingora"
version = "0.1.0"
edition = "2024"

[dependencies]
axum = { version = "0.8.4", optional = true }
tokio = { version = "1.44.2", features = ["full"], optional = true }

# pingora = "0.5.0"
pingora-core = { version = "0.5.0", features = ["rustls"], optional = true }
pingora-proxy = { version = "0.5.0", optional = true }
pingora-load-balancing = { version = "0.5.0", optional = true }
pingora-http = { version = "0.5.0", optional = true }
async-trait = { version = "0.1.52", optional = true }
log = { version = "0.4.17", optional = true }
env_logger = { version = "0.11.8", optional = true }
clap = { version = "4.5.39", features = ["derive"] }
prometheus = { version = "0.14.0", optional = true }
once_cell = "1.21.3"

[features]
server = ["axum", "tokio"]
proxy = [
    "pingora-proxy",
    "pingora-http",
    "pingora-load-balancing",
    "async-trait",
    "log",
    "env_logger",
    "pingora-core",
    "prometheus",
]

[[bin]]
name = "server"
path = "src/server.rs"
required-features = ["server"]

[[bin]]
name = "proxy"
path = "src/proxy.rs"
required-features = ["proxy"]

[hopkings2008]

I encounted the same problem, and does pingora have built-in metrics? i used the sameple from the pingora code and export the prometheus scaped server, and accessed that url, but i got nothing.

[indramal]

Any Solution?

[hopkings2008]

from pingora source codes, it seems that pingora doesn't have built-in metrics, users need to add their metrics into the related filters. Instead, pingora only provides a promethus agent.

[indramal]

@hopkings2008 so did you manage to get promethus values? Any code?

[hopkings2008]

@hopkings2008 so did you manage to get promethus values? Any code?
Currently, i didin't get the values exported from promethus provided by pingora.

[Object905]

I've just noticed similar problem after upgrading prometheus crate to 0.14. 0.13 works fine.

[indramal]

Yeah, 0.13 work fine but not for 0.14

[hopkings2008]

Yeah, 0.13 work fine but not for 0.14

after set the prometheus to 0.13, but the problem still exists and there is nothing displayed in the export url by prometheus.

[JonasCir]

Same here.

If I fetch metrics programmatically

pub fn report_metrics() -> String {
    let encoder = TextEncoder::new();
    let metric_families = prometheus::gather();
    let mut buffer = vec![];
    encoder.encode(&metric_families, &mut buffer).unwrap();
    String::from_utf8(buffer).unwrap()
}

and print it, it works. But nothing via Service::prometheus_http_service.

[v3xro]

This is more a dependency conflict than any code incompatibility I think. 0.13 and 0.14 are semver incompatible versions:

This guide uses the terms “major” and “minor” assuming this relates to a “1.0.0” release or later. Initial development releases starting with “0.y.z” can treat changes in “y” as a major release, and “z” as a minor release. “0.0.z” releases are always major changes. This is because Cargo uses the convention that only changes in the left-most non-zero component are considered incompatible.

(from https://doc.rust-lang.org/cargo/reference/semver.html)

You can't patch out semver incompatibilities in Cargo.toml so what happens when you compare the cargo tree output is that the pingora crate gets a different version of the prometheus crate from the rest of your application, hence a different registry, etc.

If you don't want to fix the prometheus version in your application you can pass version = "*" and it will take the version from pingora.