Adds OTEL instrumentation to OpenAI API server

[bbartels]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Purpose

This adds the ability to publish OTEL metrics from the openai API server

Test Plan

1. Setup required OTEL env vars
2. Run the openai api server
3. Validate metrics were published to your configured endpoint

Test Result

Metrics were correctly published

(Optional) Documentation Update

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Summary of Changes

Hello @bbartels, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces OpenTelemetry (OTEL) instrumentation to the OpenAI API server. The primary goal is to enable distributed tracing for requests processed by the server, enhancing observability and facilitating performance monitoring. This involves adding new OpenTelemetry dependencies, integrating a dedicated observability configuration throughout the engine, and setting up the FastAPI application to export traces via OTLP.

Highlights

• Dependency Management: Added several OpenTelemetry-related Python packages to requirements/common.txt, including opentelemetry-sdk, opentelemetry-api, opentelemetry-exporter-otlp, opentelemetry-semantic-conventions-ai, and opentelemetry-instrumentation-fastapi.
• Observability Configuration Integration: Introduced a new ObservabilityConfig type and integrated its retrieval across various engine components. This includes adding get_observability_config methods to AsyncLLMEngine, LLMEngine, and the MQLLMEngineClient, as well as defining it as an abstract method in the engine protocol.py.
• FastAPI OpenTelemetry Instrumentation: Implemented a setup_otel function in vllm/entrypoints/openai/api_server.py to configure OpenTelemetry tracing for the FastAPI application. This function sets up a TracerProvider, an OTLPSpanExporter (using a configurable OTLP traces endpoint), and instruments the FastAPI app using FastAPIInstrumentor.
• Dynamic Instrumentation Activation: Modified the build_app function in api_server.py to accept and utilize the ObservabilityConfig. The setup_otel function is conditionally called within build_app only if an OTLP traces endpoint is specified in the configuration, allowing for flexible enablement of tracing.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces OpenTelemetry instrumentation to the OpenAI API server, enhancing observability by exporting traces to an OTLP endpoint. The changes involve adding necessary dependencies, creating a function to set up OTEL, and integrating it into the FastAPI application. The code appears well-structured and addresses the core objective of adding OTEL support. I've provided some suggestions for improved error handling, documentation, and code clarity.

[gemini-code-assist]

This setup_otel function initializes and configures OpenTelemetry. It would be beneficial to add a docstring explaining its purpose, parameters, and any potential side effects (e.g., setting global tracer provider).

def setup_otel(app: FastAPI, observability_config: ObservabilityConfig):
    """Initializes and configures OpenTelemetry for the FastAPI application.

    Args:
        app: The FastAPI application instance.
        observability_config: The ObservabilityConfig containing OTEL settings.
    """
    from opentelemetry import trace
    from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
    from opentelemetry.sdk.trace import TracerProvider
    from opentelemetry.sdk.trace.export import BatchSpanProcessor
    from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
    from opentelemetry.sdk.resources import SERVICE_NAME, Resource

    trace.set_tracer_provider(TracerProvider(resource=Resource.create()))

    otlp_exporter = OTLPSpanExporter(endpoint=observability_config.otlp_traces_endpoint)
    trace.get_tracer_provider().add_span_processor(
        BatchSpanProcessor(otlp_exporter)
    )

    FastAPIInstrumentor().instrument_app(app)

[gemini-code-assist]

Consider adding a try-except block around the OTLPSpanExporter initialization to catch potential exceptions (e.g., network errors, invalid endpoint). This would allow for more graceful error handling and prevent the server from crashing if the OTEL exporter fails to initialize.

    try:
        otlp_exporter = OTLPSpanExporter(endpoint=observability_config.otlp_traces_endpoint)
    except Exception as e:
        logger.error(f"Failed to initialize OTLP exporter: {e}")
        return # Or raise, depending on desired behavior

[bbartels]

Just saw: #19378
@kouroshHakha What issue did customers report? This conditionally imports the instrumentation if OTLP configuration is set. Would this suffice to not cause the same issues?

[kouroshHakha]

hi @bbartels so the big issue was making open-telemetry a required vllm dependency It can cause unsolvable conflicts for downstream integrations.

From my understanding, this was the issue:

• opentelemetry-sdk>=1.30.0 // required by ray
• opentelemetry-exporter-otlp>=1.26.0 // required by vllm and used in ray-llm image
• protobuf>=4,<5 // required by ray users

It’s not easy to satisfy these constraints.

[bbartels]

Seems like this should resolve it on the ray side: ray-project/ray#53745

I'll leave this open until that is merged and a new ray version is published!

[can-anyscale]

Hi @bbartels, I believe everything in requirements/common.txt is a hard dependency and not conditionally imported. Including opentelemetry-exporter-otlp as a dependency has caused issues for our users, as recent versions require upgrading to Protobuf 5+, which many users aren't ready for.

This PR — ray-project/ray#53745 — actually also removes opentelemetry-exporter-otlp on the Ray side, in line with recent changes in the vLLM project.

TL;DR: I wouldn't recommend making opentelemetry-exporter-otlp a hard dependency for vLLM. Users with workflows that require it — and who are okay with upgrading Protobuf — can install it as needed.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @bbartels.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork