Skip to content

Support scraping metrics from target running with TLS #1190

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 22, 2025
Merged

Support scraping metrics from target running with TLS #1190

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
bb8696c
Support scraping metrics from target running with TLS
pierDipi Jul 18, 2025
5110a95
Add '--model-server-metrics-*' flags to Helm values and EPP template
pierDipi Jul 21, 2025
40e19ae
Merge branch 'main' into https-scraping
pierDipi Jul 22, 2025
54dcf0d
Update config/charts/inferencepool/values.yaml
kfswain Jul 22, 2025
d26fb0d
Update cmd/epp/runner/runner.go
kfswain Jul 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 36 additions & 11 deletions cmd/epp/runner/runner.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,11 @@ package runner

import (
"context"
"crypto/tls"
"errors"
"flag"
"fmt"
"net/http"
"net/http/pprof"
"os"

Expand Down Expand Up @@ -138,7 +140,9 @@ var (

modelServerMetricsPort = flag.Int("model-server-metrics-port", 0, "Port to scrape metrics from pods. "+
"Default value will be set to InferencePool.Spec.TargetPortNumber if not set.")
modelServerMetricsPath = flag.String("model-server-metrics-path", "/metrics", "Path to scrape metrics from pods")
modelServerMetricsPath = flag.String("model-server-metrics-path", "/metrics", "Path to scrape metrics from pods")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Do you mind adding these flags (with the same defaults) to the helm chart? Example:

gateway-api-inference-extension/config/charts/inferencepool/templates/epp-deployment.yaml

Line 41 in 6cf8f31

- "--enable-pprof={{ .Values.inferenceExtension.enablePprof }}"

Thanks!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the feedback Kellen, done!

modelServerMetricsScheme = flag.String("model-server-metrics-scheme", "http", "Scheme to scrape metrics from pods")
modelServerMetricsHttpsInsecureSkipVerify = flag.Bool("model-server-metrics-https-insecure-skip-verify", true, "When using 'https' scheme for 'model-server-metrics-scheme', configure 'InsecureSkipVerify' (default to true)")

setupLog = ctrl.Log.WithName("setup")
)
Expand Down Expand Up @@ -169,13 +173,15 @@ func (r *Runner) WithSchedulerConfig(schedulerConfig *scheduling.SchedulerConfig
func bindEnvToFlags() {
// map[ENV_VAR]flagName – add more as needed
for env, flg := range map[string]string{
"GRPC_PORT": "grpc-port",
"GRPC_HEALTH_PORT": "grpc-health-port",
"MODEL_SERVER_METRICS_PORT": "model-server-metrics-port",
"MODEL_SERVER_METRICS_PATH": "model-server-metrics-path",
"DESTINATION_ENDPOINT_HINT_KEY": "destination-endpoint-hint-key",
"POOL_NAME": "pool-name",
"POOL_NAMESPACE": "pool-namespace",
"GRPC_PORT": "grpc-port",
"GRPC_HEALTH_PORT": "grpc-health-port",
"MODEL_SERVER_METRICS_PORT": "model-server-metrics-port",
"MODEL_SERVER_METRICS_PATH": "model-server-metrics-path",
"MODEL_SERVER_METRICS_SCHEME": "model-server-metrics-scheme",
"MODEL_SERVER_METRICS_HTTPS_INSECURE_SKIP_VERIFY": "model-server-metrics-https-insecure-skip-verify",
"DESTINATION_ENDPOINT_HINT_KEY": "destination-endpoint-hint-key",
"POOL_NAME": "pool-name",
"POOL_NAMESPACE": "pool-namespace",
// durations & bools work too; flag.Set expects the *string* form
"REFRESH_METRICS_INTERVAL": "refresh-metrics-interval",
"SECURE_SERVING": "secure-serving",
Expand Down Expand Up @@ -235,10 +241,26 @@ func (r *Runner) Run(ctx context.Context) error {
return err
}
verifyMetricMapping(*mapping, setupLog)

var metricsHttpClient *http.Client
if *modelServerMetricsScheme == "https" {
metricsHttpClient = &http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{
InsecureSkipVerify: *modelServerMetricsHttpsInsecureSkipVerify,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how is the CA added to the TLS context? Is it intended to be added to the default certificate roots in the image?
Without adding the CA to the allowed list - won't you be forced to always use insecure-skip-verify set to true?

Copy link
Contributor

@ahg-g ahg-g Jul 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, we should actually remove the flag for now and set this to true

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elevran agreed that this is suboptimal. Unfortunately this seems to be blocking downstream & is being pushed to be included in a v0.5.1 patch. WE will move forward with this, but would like @pierDipi to create an issue to track CA validation to help prevent MITM vulnerabilites.

Copy link
Contributor Author

@pierDipi pierDipi Jul 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Go, we can inject the default loaded CA certs file and dirs paths via env variables https://go.dev/src/crypto/x509/root_unix.go or use well-known files/directories https://go.dev/src/crypto/x509/root_linux.go:

	// certFileEnv is the environment variable which identifies where to locate
	// the SSL certificate file. If set this overrides the system default.
	certFileEnv = "SSL_CERT_FILE"

	// certDirEnv is the environment variable which identifies which directory
	// to check for SSL certificate files. If set this overrides the system default.
	// It is a colon separated list of directories.
	// See https://www.openssl.org/docs/man1.0.2/man1/c_rehash.html.
	certDirEnv = "SSL_CERT_DIR"
// Possible certificate files; stop after finding one.
var certFiles = []string{
	"/etc/ssl/certs/ca-certificates.crt",                // Debian/Ubuntu/Gentoo etc.
	"/etc/pki/tls/certs/ca-bundle.crt",                  // Fedora/RHEL 6
	"/etc/ssl/ca-bundle.pem",                            // OpenSUSE
	"/etc/pki/tls/cacert.pem",                           // OpenELEC
	"/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", // CentOS/RHEL 7
	"/etc/ssl/cert.pem",                                 // Alpine Linux
}

// Possible directories with certificate files; all will be read.
var certDirectories = []string{
	"/etc/ssl/certs",     // SLES10/SLES11, https://golang.org/issue/12139
	"/etc/pki/tls/certs", // Fedora/RHEL
}

func init() {
	if goos.IsAndroid == 1 {
		certDirectories = append(certDirectories,
			"/system/etc/security/cacerts",    // Android system roots
			"/data/misc/keychain/certs-added", // User trusted CA folder
		)
	}
}

do we still want to add the explicit CA path flags or we can leverage built-in/standard methods to inject custom CA bundles?

Copy link
Contributor

@elevran elevran Jul 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pierDipi as far as I can tell, this configures Go for directory locations where CA files are located.
I think the question is which component is responsible for adding the relevant CA(s) to the EPP Pods. Without orchestrating that, we would need to skip verification on the client (EPP) side since there's no assurance the CA signing the vLLM certs is known to the EPP.
It's a process/deployment concern, not a technical limitation per-se.
Agree it's better handled in a separate issue/PR and we use insecure-skip-verify=true for now to get the encryption enabled.

},
},
}
} else {
metricsHttpClient = http.DefaultClient
}

pmf := backendmetrics.NewPodMetricsFactory(&backendmetrics.PodMetricsClientImpl{
MetricMapping: mapping,
ModelServerMetricsPort: int32(*modelServerMetricsPort),
ModelServerMetricsPath: *modelServerMetricsPath,
MetricMapping: mapping,
ModelServerMetricsPort: int32(*modelServerMetricsPort),
ModelServerMetricsPath: *modelServerMetricsPath,
ModelServerMetricsScheme: *modelServerMetricsScheme,
Client: metricsHttpClient,
}, *refreshMetricsInterval)

datastore := datastore.NewDatastore(ctx, pmf)
Expand Down Expand Up @@ -421,6 +443,9 @@ func validateFlags() error {
if *configText != "" && *configFile != "" {
return fmt.Errorf("both the %q and %q flags can not be set at the same time", "configText", "configFile")
}
if *modelServerMetricsScheme != "http" && *modelServerMetricsScheme != "https" {
return fmt.Errorf("unexpected %q value for %q flag, it can only be set to 'http' or 'https'", *modelServerMetricsScheme, "model-server-metrics-scheme")
}

return nil
}
Expand Down
3 changes: 3 additions & 0 deletions config/charts/inferencepool/templates/epp-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,9 @@ spec:
- "config/{{ .Values.inferenceExtension.pluginsConfigFile }}"
# https://pkg.go.dev/flag#hdr-Command_line_flag_syntax; space is only for non-bool flags
- "--enable-pprof={{ .Values.inferenceExtension.enablePprof }}"
- "--model-server-metrics-path={{ .Values.inferenceExtension.modelServerMetricsPath }}"
- "--model-server-metrics-scheme={{ .Values.inferenceExtension.modelServerMetricsScheme }}"
- "--model-server-metrics-https-insecure-skip-verify={{ .Values.inferenceExtension.modelServerMetricsHttpsInsecureSkipVerify }}"
{{- if eq (.Values.inferencePool.modelServerType | default "vllm") "triton-tensorrt-llm" }}
- --total-queued-requests-metric
- "nv_trt_llm_request_metrics{request_type=waiting}"
Expand Down
3 changes: 3 additions & 0 deletions config/charts/inferencepool/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ inferenceExtension:
extProcPort: 9002
env: {}
enablePprof: true # Enable pprof handlers for profiling and debugging
modelServerMetricsPath: "/metrics"
modelServerMetricsScheme: "http"
modelServerMetricsHttpsInsecureSkipVerify: true
# This is the plugins configuration file.
pluginsConfigFile: "default-plugins.yaml"
# pluginsCustomConfig:
Expand Down
13 changes: 8 additions & 5 deletions pkg/epp/backend/metrics/metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,9 +37,12 @@ const (
)

type PodMetricsClientImpl struct {
MetricMapping *MetricMapping
ModelServerMetricsPort int32
ModelServerMetricsPath string
MetricMapping *MetricMapping
ModelServerMetricsPort int32
ModelServerMetricsPath string
ModelServerMetricsScheme string

Client *http.Client
}

// FetchMetrics fetches metrics from a given pod, clones the existing metrics object and returns an updated one.
Expand All @@ -49,7 +52,7 @@ func (p *PodMetricsClientImpl) FetchMetrics(ctx context.Context, pod *backend.Po
if err != nil {
return nil, fmt.Errorf("failed to create request: %v", err)
}
resp, err := http.DefaultClient.Do(req)
resp, err := p.Client.Do(req)
if err != nil {
return nil, fmt.Errorf("failed to fetch metrics from %s: %w", pod.NamespacedName, err)
}
Expand All @@ -73,7 +76,7 @@ func (p *PodMetricsClientImpl) getMetricEndpoint(pod *backend.Pod, targetPortNum
if p.ModelServerMetricsPort == 0 {
p.ModelServerMetricsPort = targetPortNumber
}
return fmt.Sprintf("http://%s:%d%s", pod.Address, p.ModelServerMetricsPort, p.ModelServerMetricsPath)
return fmt.Sprintf("%s://%s:%d%s", p.ModelServerMetricsScheme, pod.Address, p.ModelServerMetricsPort, p.ModelServerMetricsPath)
}

// promToPodMetrics updates internal pod metrics with scraped Prometheus metrics.
Expand Down
9 changes: 8 additions & 1 deletion pkg/epp/backend/metrics/metrics_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ package metrics
import (
"context"
"errors"
"net/http"
"reflect"
"strconv"
"strings"
Expand Down Expand Up @@ -495,7 +496,13 @@ func TestFetchMetrics(t *testing.T) {
},
}
existing := &MetricsState{}
p := &PodMetricsClientImpl{ModelServerMetricsPort: 9999, ModelServerMetricsPath: "/metrics"} // No MetricMapping needed for this basic test
// No MetricMapping needed for this basic test
p := &PodMetricsClientImpl{
ModelServerMetricsScheme: "http",
ModelServerMetricsPort: 9999,
ModelServerMetricsPath: "/metrics",
Client: http.DefaultClient,
}

_, err := p.FetchMetrics(ctx, pod, existing, 9999) // Use a port that's unlikely to be in use
if err == nil {
Expand Down