[Feature][v1]: Add metrics support

[rickyyx]

🚀 The feature, motivation and pitch

We should also be feature parity on metrics with most of available stats if possible. On a high level:

1. [P0] Support system and requests stats logging
2. [P0] Support metric export to prometheus.
3. [P1] Support or deprecate all metrics from V0
4. [P1] Allow users to define their own prometheus client and other arbitrary loggers.
5. [P2] Make it work with tracing too (there's some request level stats that tracing needs, like queue time, ttft). These request level metric should be possible to be surfaced in v1 too.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[rickyyx]

Opening the issue to track and collab - in case someone else is already looking into this.

[rickyyx]

Prototype in #10651

[markmc]

I thought it was about time to update on the latest status of this and note some TODOs.

Status

The v1 engine frontend API server now has a Prometheus-compatible `/metrics' endpoint.

The following PRs should merge soon:

• [V1][Metrics] Add request_success_total counter, labelled with finish reason #12579
• [V1][Metrics] Add GPU prefix cache hit rate % gauge #12592
• [V1][Metrics] Add several request timing histograms #12644

Which will mean we support the following metrics:

• vllm:num_requests_running (Gauge)
• vllm:num_requests_waiting (Gauge)
• vllm:gpu_cache_usage_perc (Gauge)
• vllm:prompt_tokens_total (Counter)
• vllm:generation_tokens_total (Counter)
• vllm:request_success_total (Counter)
• vllm:request_prompt_tokens (Histogram)
• vllm:request_generation_tokens (Histogram)
• vllm:time_to_first_token_seconds (Histogram)
• vllm:time_per_output_token_seconds (Histogram)
• vllm:e2e_request_latency_seconds (Histogram)
• vllm:request_queue_time_seconds (Histogram)
• vllm:request_inference_time_seconds (Histogram)
• vllm:request_prefill_time_seconds (Histogram)
• vllm:request_decode_time_seconds (Histogram)

Also, note that - vllm:gpu_prefix_cache_queries and vllm:gpu_prefix_cache_hits (Counters) replaces vllm:gpu_prefix_cache_hit_rate (Gauge).

These are most of the metrics used by the example Grafana dashboard, with the exception of:

• vllm:num_requests_swapped (Gauge)
• vllm:cpu_cache_usage_perc (Gauge)
• vllm:request_max_num_generation_tokens (Histogram)

Additionally, these are other metrics supported by v0, but not yet by v1:

• vllm:num_preemptions_total (Counter)
• vllm:cache_config_info (Gauge)
• vllm:lora_requests_info (Gauge)
• vllm:cpu_prefix_cache_hit_rate (Gauge)
• vllm:tokens_total (Counter)
• vllm:iteration_tokens_total (Histogram)
• vllm:time_in_queue_requests (Historgram)
• vllm:model_forward_time_milliseconds (Histogram
• vllm:model_execute_time_milliseconds (Histogram)
• vllm:request_params_n (Histogram)
• vllm:request_params_max_tokens (Histogram)
• vllm:spec_decode_draft_acceptance_rate (Gauge)
• vllm:spec_decode_efficiency (Gauge)
• vllm:spec_decode_num_accepted_tokens_total (Counter)
• vllm:spec_decode_num_draft_tokens_total (Counter)
• vllm:spec_decode_num_emitted_tokens_total (Counter)

Next Steps

• Merge [V1][Metrics] Add request_success_total counter, labelled with finish reason #12579
• Merge [V1][Metrics] Add GPU prefix cache hit rate % gauge #12592
• Merge [V1][Metrics] Add several request timing histograms #12644
• Make sure --disable-log-stats is disabling everything it can
• Go through the remaining v0 metrics (see design doc)
• Merge [V1][Metrics] Add iteration_tokens_total histogram from V0 #13288
• Merge [Metrics] Add --show-hidden-metrics-for-version CLI arg #13295
• Merge [V1][Metrics] Support vllm:cache_config_info #13299
• Merge [V1][Metrics] Implement vllm:lora_requests_info metric #13504
• Update design doc with preemption diagrams from [V1][Metrics] Handle preemptions #13169
• Merge [V1][Metrics] Handle preemptions #13169
• Update Grafana dashboard to work with v1