Exposing TLS certificate metrics

This is a question/feature request.

For caddy-prometheus (github.com/miekg/caddy-prometheus) I would like to expose metrics from the certs caddy has configured. Like time left per cert, failures to update, etc. etc.

Is there an API for this, should caddy expose some of these bits or should I hack in into ACME (? believe that is used for handling LE comms)?

@miekg

Hello @miekg,

this sounds like a great idea!

I believe the best approach would be to create an API in github.com/xenolf/lego/acme -- It might be useful for other projects using lego/acme

Basically you should "hack in into ACME" 😄

__

On the other side, I believe it would be easier to create those metics directly in Caddy and expose them to caddy-prometheus

[ Quoting <notifications@github.com> in "Re: [mholt/caddy] Exposing TLS cert..." ]

Hello @miekg, this sounds like a great idea! I believe the best approach would be to expose those metrics in `github.com/xenolf/lego/acme` -- It might be useful for other projects using `lego/acme` Basically you should "hack in into ACME" 😄

Interesting thought. I'll look into that, there could either be an API in ACME, or directly expose prometheus metrics. I'll open an issue there.

On the other side, I believe it would be easier to create those metics directly in Caddy and expose them to `caddy-prometheus`

But how do I get access to the information that those metrics will expose? /Miek

…

-- Miek Gieben

But how do I get access to the information that those metrics will expose?

You could import caddy/caddytls in caddy-prometheus and use those metrics -- I have not really thought deeply about this ..........

I just posted some ideas I had 😄, my raw thoughts.

Interesting thought. I'll look into that, there could either be an API in ACME,
or directly expose prometheus metrics. I'll open an issue there.

Awesome 😄

I think we can do something like this; Caddy can emit events but right now it only emits one (startup) because we're adding events based on need. I don't have a page on the wiki yet explaining how to hook into events (I'll do that soon) but it's very easy; you can look at the caddy-service plugin and see how it does it.

Although I think we'll need to change the signature of event hooks to pass in some information. Shouldn't be too hard since only that plugin uses event hooks; I'll talk to the authors about getting it updated.

I think this is better done in Caddy than lego.

Interesting. What new events would we need? As I alluded to above, qps to let's encrypt, possible errors returned from the let's encrypt, TLS cert properties (don't know what is interesting there).

We could add a lot of events around TLS management. Caddy scans loaded certs every 12 hours for renewal, or every hour for OCSP stapling, those are events (although maybe uninteresting, and not needed to emit). When Caddy starts renewing a certificate, or finishes successfully or has an error, those are events. When it updates OCSP staples, that's an event. When it obtains a new certificate with on-demand TLS, that's an event. When it loads a certificate from disk, that's an event. By emitting these with some information attached (currently infeasible but we can change that) you should be able to get what you'd need, I think?

QPS to Let's Encrypt specifically doesn't make much sense since requests to LE are few and far between.

Edit: If you want really fine-grained information, such as every low-level network interaction that lego makes in order to validate a challenge, that'll have to be built into lego, of course, and I dunno how that would go. But for everything at or above "Obtain a certificate" or "renew a certificate", Caddy can emit those.

@miekg

@miekg If this is still something you're interested in, I think you should join our discussion in the #telemetry channel in Slack. I think that will be the best fit for this feature, since we expect telemetry to be exportable in various formats or directly to external monitoring services.

Either that or we overhaul logging to be able to tee to different outputs.

@hairyhenderson

/cc @hairyhenderson

@mholt

@mholt
Hello ! 😃
Any recent update on this one ?
It would be awesome to have the aforementioned metrics integrated !

Just adding a top-up request for TLS certificate metrics; specifically I'm interested in metrics on ACME renewals and particularly renewal failures. The goal here is to get metrics on how many times we are renewing/issuing within a given window to see where we are in relation to LetsEncrypt rate limits, and additionally any metrics on failed renewals would be useful, since failed validations count towards the new order limits.

It'd be great if we can collaboratively create the list of metrics labels and descriptions based on what you, as users, need to track. Based on that, we can assess wiring up the collectors.

Sure, I'm happy to kick us off, here's some examples of things I think I'd find useful.

I'd group it all under 'certs', try to correlate to the certmagic events?

# HELP caddy_certs_obtain_total Total certificates obtained.
# TYPE caddy_certs_obtain_total counter
caddy_certs_obtain_total{issuer="acme",server="srv3","action":"new", "directory": "https://acme-staging-v02.api.letsencrypt.org/directory"} 1
caddy_certs_obtain_total{issuer="acme",server="srv3","action":"renewal", "directory": "https://acme-staging-v02.api.letsencrypt.org/directory"} 1

# HELP caddy_certs_obtain_error_total Total certificate errors.
# TYPE caddy_certs_obtain_error_total counter
caddy_certs_obtain_error_total{issuer="acme",server="srv3","action":"new", "reason": "validation-failed", "directory": "https://acme-staging-v02.api.letsencrypt.org/directory"} 1
caddy_certs_obtain_error_total{issuer="acme",server="srv3","action":"renewal", "reason": "rate-limit", "directory": "https://acme-staging-v02.api.letsencrypt.org/directory"} 1

# HELP caddy_certs_obtain_inflight In-flight certificates being obtained.
# TYPE caddy_certs_obtain_inflight gauge
caddy_certs_obtain_inflight{issuer="acme",server="srv3","action":"new", "directory": "https://acme-staging-v02.api.letsencrypt.org/directory"} 1
caddy_certs_obtain_inflight{issuer="internal",server="srv3","action":"renewal"} 1

# HELP caddy_certs_remaining_seconds_at_renewal The remaining time on certificates at successful renewal (if this is skewing to quite late it means we're not hitting that 1/3 target)
# TYPE caddy_certs_remaining_seconds_at_renewal histogram
caddy_certs_remaining_seconds_at_renewal_bucket{issuer="acme",le="0.005"} 5674
caddy_certs_remaining_seconds_at_renewal_bucket{issuer="acme",le="0.01"} 5849
caddy_certs_remaining_seconds_at_renewal_bucket{issuer="acme",le="0.025"} 13168
caddy_certs_remaining_seconds_at_renewal_bucket{issuer="acme",le="0.05"} 17458
caddy_certs_remaining_seconds_at_renewal_bucket{issuer="acme",le="0.1"} 21198
caddy_certs_remaining_seconds_at_renewal_bucket{issuer="acme",le="0.25"} 23510
caddy_certs_remaining_seconds_at_renewal_bucket{issuer="acme",le="0.5"} 23721
caddy_certs_remaining_seconds_at_renewal_bucket{issuer="acme",le="1"} 23753
caddy_certs_remaining_seconds_at_renewal_bucket{issuer="acme",le="2.5"} 23793
caddy_certs_remaining_seconds_at_renewal_bucket{issuer="acme",le="5"} 23850
caddy_certs_remaining_seconds_at_renewal_bucket{issuer="acme",le="10"} 23923
caddy_certs_remaining_seconds_at_renewal_bucket{issuer="acme",le="+Inf"} 23998
caddy_http_request_duration_seconds_sum{issuer="acme"} 3877.6205545949847
caddy_http_request_duration_seconds_count{issuer="acme"} 23998

Having just gone through an incident with some on-demand TLS issues.
The issue we had there required us to notice a bunch of new certificates being issued which, when looking on the ask backend is a little
amongst the metics I would like to see are

# caddy_on_demand_tls_ask_total
# How many `ask` operations have happened, grouped by the thing we're asking and the result`
# Not sure quite what the set of results should look like -- might want to be the http status code instead?
caddy_on_demand_tls_ask_total{srv="conf-srv",upstream="http://backed:8080/cert-check",result="(true|false|timeout|conn-error)"}

# caddy_on_demand_tls_obtain_total
# Record of all attempts to issue certs. Issuer is a key filtering label; I've included the account and
# storage backend here to with the expectation that they're not going to add to the cardinality in most configurations.
# (Does caddy support multiple sotrage configurations in different scopes?)
# Otherwise, this is mostly the same as alistairjevans' metrics, except grouping errors and successes as a label
caddy_on_deman_tls_obtain_total{srv="conf-srv",issuer="https://acme-v02.api.letsencrypt.org/directory",account="my-el-account@example.org",store="...",action="(issue|renew|revoke)",result="(success|[various error states])"}

# There may be some interest in a histogram of how long it took things to issue -- but that is much more passing curiosity.
caddy_on_demand_tls_latency_bucket{...}

# And, as alistairjevans suggests, perhaps something around renewals
caddy_on_demand_tls_remaining_seconds_bucket

@javajawa

Hello everyone! 👋

I just opened a PR that addresses this feature request: #7442

It introduces two new counter metrics to the caddytls module:

caddy_tls_obtain_total (labeled by issuer and result)
caddy_tls_on_demand_ask_total (labeled by result)

@javajawa and @alistairjevans, thank you for your recent contributions regarding the metric names and labels! I've tried to align the fix with your suggestions whenever possible to cover both certificate issuance and on-demand abuse tracking.

Comments are welcome!

@javajawa

+1 for renewal-failure metrics, with a production use case. We run Caddy with on_demand_tls for thousands of customer domains. A customer's DNS changed (proxied through a CDN), and HTTP-01/TLS-ALPN-01 validation started failing. Caddy retried renewal for 25 days until Let's Encrypt's consecutive-failure limit paused the identifier, and the cert expired. Nothing in /metrics showed it; we found it in tls.renew logs after the outage.

What would have caught it:

caddy_tls_obtain_total{issuer, result="success|failure", error_type}, where error_type is something like rate_limited, challenge_failed or other. Successes alone (as in feat: add prometheus metrics for TLS issuance and on-demand asks #7442) aren't enough.
caddy_tls_managed_cert_earliest_expiry_timestamp_seconds, a single gauge with no per-name label, safe with on-demand.
caddy_tls_on_demand_ask_total{result}, as proposed by @javajawa.
A per-identifier label would be ideal but is likely unsafe with on-demand TLS, so aggregate counters are fine; we use logs for per-domain detail.

Bounded counters for obtain and renewal outcomes, an error category such as challenge_failed or rate_limited, and one gauge for the earliest managed-certificate expiry. Do not use certificate names, account addresses or other unbounded values as labels; on-demand TLS makes those attacker-controlled. on_demand_ask_total{result} is also reasonable. #7442 does not meet the requested scope and has been closed.

@steadytao

@steadytao I'd like to take this on, following the scope above. Before writing any code, here's the metric contract I have in mind, so it can be corrected first:

caddy_tls_certificate_operations_total{operation, issuer, result, error}
    operation: "obtain" | "renew"
    result:    "success" | "failure"
    error:     "" on success; on failure "challenge_failed" | "rate_limited" | "other"
caddy_tls_on_demand_ask_total{result}
    result: "allowed" | "denied" | "error"
caddy_tls_managed_certificates_earliest_expiry_timestamp_seconds   (gauge, no labels)

The counters are fed by the cert_obtained and cert_failed events certmagic already emits through TLS.onEvent, so there's no polling and no change to issuance itself.
issuer is the issuer key from the event, so its values come from the configured issuers, not from clients. On failure it's the last issuer tried, whose error is the one returned. If you'd rather leave issuer out entirely, that's fine too.
error comes from the ACME problem type: rateLimited is rate_limited; connection, dns, unauthorized, incorrectResponse, tls and caa are challenge_failed; anything else, including non-ACME errors, is other.
The ask counter sits where the automation policy calls the permission module, so it covers any permission module, not only the HTTP one: no error is allowed, ErrPermissionDenied is denied, anything else is error.
No identifier, SAN, account or ask URL appears in any label.

Two questions:

certmagic's cache doesn't expose a way to walk the managed certificates. For the expiry gauge, would you prefer a small exported method in certmagic (for example, the earliest expiry among managed certificates), or tracking it in caddytls from the certificate events?
Should these be registered only when metrics are enabled, like the HTTP server metrics, or always, since they're a handful of series?

(I used AI assistance, Claude, to research and draft this; I've reviewed it and stand behind it.)

Looks fairly sensible. Use error_type rather than error and keep issuer limited to the configured issuer key. The expiry gauge should come from a small aggregate method in CertMagic rather than reconstructed events. Also note that ShouldEmit currently suppresses OnEvent so metrics must enable the relevant events. Register the bounded collectors unconditionally.

Navigation Menu

Repository navigation

Exposing TLS certificate metrics #1683

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Exposing TLS certificate metrics #1683

Description

Activity

elcore commented on May 23, 2017

miekg commented on May 23, 2017

elcore commented on May 23, 2017

elcore commented on May 24, 2017

mholt commented on May 26, 2017

miekg commented on May 26, 2017

mholt commented on May 26, 2017

mholt commented on Mar 27, 2018

mholt commented on Sep 17, 2020

loss commented on Jul 26, 2024

alistairjevans commented on Sep 18, 2024

mohammed90 commented on Sep 18, 2024

alistairjevans commented on Sep 18, 2024

javajawa commented on Jan 17, 2025

pauloappbr commented on Jan 17, 2026

davidxia commented on Oct 3, 2026

steadytao commented on Oct 4, 2026

IslamElsayed commented on Oct 9, 2026

steadytao commented on Oct 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions