OpenTelemetry instrumentation with Sentry GRPCIntegration raises an exception

[danielrozyckizego]

How do you use Sentry?

Sentry Saas (sentry.io)

Version

2.27.0

Steps to Reproduce

1. Install sentry-sdk[grpcio] in version 2.27.0
2. Install opentelemetry-instrumentation-grpc==0.53b0
3. Setup sentry with integrations=[GRPCIntegration()]
4. Run gRPC server with instrumentation like opentelemetry-instrument python -m app.grpc_api.grpc_server

Expected Result

Server runs without an issue and Sentry GRPCIntegration works

Actual Result

An exception is raised in opentelemetry-instrument

  File "/Users/Tosuto/PycharmProjects/service/app/grpc_api/grpc_server.py", line 91, in start_server
    self.grpc_server = grpc.aio.server()
                       ^^^^^^^^^^^^^^^^^
  File "/Users/Tosuto/.pyenv/versions/3.12.7/envs/3.12.7/lib/python3.12/site-packages/sentry_sdk/integrations/grpc/__init__.py", line 131, in patched_aio_server
    return func(*args, interceptors=interceptors, **kwargs)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/Tosuto/.pyenv/versions/3.12.7/envs/3.12.7/lib/python3.12/site-packages/opentelemetry/instrumentation/grpc/__init__.py", line 391, in server
    kwargs["interceptors"].insert(
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'tuple' object has no attribute 'insert'

It appears that Sentry in _wrap_async_server passes tuple when opentelemetry-instrumentation expects list, see:

# Sentry
def _wrap_async_server(func: Callable[P, AsyncServer]) -> Callable[P, AsyncServer]:
    """Wrapper for asynchronous server."""

    @wraps(func)
    def patched_aio_server(  # type: ignore
        *args: P.args,
        interceptors: Optional[Sequence[grpc.ServerInterceptor]] = None,
        **kwargs: P.kwargs,
    ) -> Server:
        server_interceptor = AsyncServerInterceptor()
        interceptors = (server_interceptor, *(interceptors or [])) # uses Tuple here
        return func(*args, interceptors=interceptors, **kwargs)  # type: ignore

    return patched_aio_server  # type: ignore

# OpenTelemetry
            if "interceptors" in kwargs:
                # add our interceptor as the first
                kwargs["interceptors"].insert( # Expects list here but Sentry already patched it with tuple
                    0,
                    aio_server_interceptor(
                        tracer_provider=tracer_provider, filter_=self._filter
                    ),
                )
            else:
                kwargs["interceptors"] = [
                    aio_server_interceptor(
                        tracer_provider=tracer_provider, filter_=self._filter
                    )
                ]
            return self._original_func(*args, **kwargs)

And that's kinda strange because in non async wraps, Sentry uses list, see:

def _wrap_sync_server(func: Callable[P, Server]) -> Callable[P, Server]:
    """Wrapper for synchronous server."""

    @wraps(func)
    def patched_server(  # type: ignore
        *args: P.args,
        interceptors: Optional[Sequence[grpc.ServerInterceptor]] = None,
        **kwargs: P.kwargs,
    ) -> Server:
        interceptors = [ 
            interceptor
            for interceptor in interceptors or []
            if not isinstance(interceptor, ServerInterceptor)
        ] 
        server_interceptor = ServerInterceptor()
        interceptors = [server_interceptor, *(interceptors or [])] # Sentry uses list here
        return func(*args, interceptors=interceptors, **kwargs)  # type: ignore

    return patched_server  # type: ignore

[szokeasaurusrex]

Hi @danielrozyckizego – I attempted to reproduce the error you observed using the instructions you provided, but unfortunately, I was unable to get the same error that you saw.

For your convenience, I have created a repository with the code I used to try to reproduce the problem, along with the steps I followed in the README.

I would appreciate if you could please take a look at my repository and let me know what I would need to change in order to get the error you observed (you can also fork or open a PR against the repository if that is easier).

Thank you!

[danielrozyckizego]

Hi @danielrozyckizego – I attempted to reproduce the error you observed using the instructions you provided, but unfortunately, I was unable to get the same error that you saw.

For your convenience, I have created a repository with the code I used to try to reproduce the problem, along with the steps I followed in the README.

I would appreciate if you could please take a look at my repository and let me know what I would need to change in order to get the error you observed (you can also fork or open a PR against the repository if that is easier).

Thank you!

Hey @szokeasaurusrex, thank you for looking into this!
Your code doesn't reproduce the problem because it's written synchronously, the issue I described appears only in async context with grpc.aio.server() -- I could've been more explicit about it, sorry.

Synchronous version works perfectly fine, the grpc.aio.server() is the problem. Sentry is patching aio_server interceptors as tuple (should be list) for async_server as showed in the issue.