GraphQLOperationTrace is never created, resulting in partial instrumentation

[Doctor-God]

Description

When any graphql query is executed, the following warning appears on the agent logs:
newrelic.hooks.framework_graphql WARNING - Runtime instrumentation warning. GraphQL operation found without active GraphQLOperationTrace.
What ends up happening is the graphql requests are not properly identified and don't have any of the expected fields, like operation_name.

Expected Behavior

GraphQL calls should be correctly instrumented.

Troubleshooting

newrelic.hooks.framework_graphql WARNING - Runtime instrumentation warning. GraphQL operation found without active GraphQLOperationTrace.

Steps to Reproduce

I tried auto instrumentation both with the "newrelic-admin run-program" and by wrapping my wsgi application, as per docs.
I monkey-patched both the hook framework_graphql and graphql-core and noticed while wrap_graphql_impl (the wrapper that creates the GraphQLOperationTrace) was applied through instrument_graphql, it is never called. graphql-core's graphql_impl is also never called when executing a query or mutation.
The trace that is found on wrap_execute_operation and results in the aforementioned warning is a FunctionTrace.
I wonder if the auto-instrumentation is wrapping the wrong function?

Your Environment

Python 3.12.5
Agent 10.15.0
django 5.2.4
graphql-core 3.2.6
graphene 3.4.3
graphene-django 3.2.3
gunicorn 23.0.0

We have somewhat of a legacy system which recently migrated to Graphene 3. We might have some custom overrides on the standard lib required to maintain our API backwards-compatible, but none of them seem to be related to the mentioned functions.

Additional context

I tried search for others with a similar problems and found the following support threads:
https://support.newrelic.com/s/hubtopic/aAXPh0000001lcnOAA/graphqloperationtrace-in-logs
https://support.newrelic.com/s/hubtopic/aAX8W0000008cXLWAY/python-warnings-regarding-graphql-trace
The latter seems to be the same problem, but the solution provided was simply to silence the warning logs, which is not what I'm looking for.

[hmstepanek]

Often when this happens it's because the library on top of GraphQL is not using graphql-core's graphql_impl() but rather implementing their own from scratch which we do not have instrumentation for. I'm going to mark this as a feature request for us to add support for graphene-django.

[lrafeei]

I have instrumented the graphene-django library on this branch:

https://github.com/newrelic/newrelic-python-agent/tree/graphene-django-instrumentation

Could you see if that resolves your issue? You can install the branch with the following command: pip install git+https://github.com/newrelic/newrelic-python-agent.git@graphene-django-instrumentation

[Doctor-God]

Hey @lrafeei, I've briefly tested the provided branch and it seems to work a lot better now!

There are two things I've noticed (take them with a grain of salt, these might be due to my inexperience with New Relic):

1. Were the traces supposed to have new attributes? Besides the generated name (which allows me to now find stuff properly, helping a lot!), nothing else gets added. I would expected to have attributes such as operation_name, to allow for better querying on dashboards.
2. Some traces still get classified (if that is even the term) weidly. For example, when executing the query queryName, I sometimes get the trace name query/queryName/queryName.edges.node, when I would expect query/queryName/queryName. Most queries and mutation seem to work as expected.

[lrafeei]

Thanks for testing this out! I added tests on my end but clearly I missed some scenarios...

1. Yikes, there are definitely supposed to be some operation attributes. operation_type and operation_query should always be in the span attributes and operation_name will be included as a span attribute if there is a name given to the query. If no operation name is given, it should be dropped from the attribute list (or you might see it as <anonymous>). I'll look into why these are not being seen.

2. The transaction naming for GraphQL can be a bit confusing. Basically, the high level naming convention for a transaction is {operation_type}/{operation_name}/{deepest_unique_path} --so if you had a query like this one:

query QueryWithDeepPath {
    libraries {
        books {
            author {
                name
            }
        }
    }
}

The deepest unique path would be "libraries.books.author.name" so the transaction name would be query/QueryWithDeepPath/libraries.books.author.name

query {
    libraries {
        books {
            title
            author {
                name
            }
        }
    }
}

In this example, the deepest unique path would be "libraries.books" so the transaction name would be query/<anonymous>/libraries.books (There's no query name here, so it gets filled in with <anonymous>)

(More info here on that)

[lrafeei]

@Doctor-God silly question for you, how are you querying for the attributes? This is what I am using in my APM:

FROM Span SELECT name, graphql.field.name, graphql.field.parentType, graphql.field.path, graphql.field.returnType, graphql.operation.name, graphql.operation.type, * WHERE name LIKE '%GrapheneDjango%'

I see all of them in all my tests and was wondering if there was some use case I missed. Thanks!

[Doctor-God]

@lrafeei Hi again. Thanks for all the help!

When looking at traces on the dashboard, I looked at attributes from the root span that was shown, which is not the one containing the relevant attributes. Seems weird to me that the root span has the name of a graphql span, but does not have graphql attributes, but I presume that's just how NR works.

The naming convention for graphql spans make sense, thanks for the docs explaining it.

About attributes again, I have a specific scenario which might or not be possible to solve:
We provide our GraphQL API directly to our clients, and they may name queries weirdly, like

query randomMeaninglessName{
    libraries {
        books {
            title
            author {
                name
            }
        }
    }
}

When querying spans, it would be a little annoying to find such queries, because operation_name would be "randomMeaninglessName", when I would prefer it to be "libraries".

I guess this might be a limitation with how GraphQL works, since a single query may be composed of mutiple "sub-queries", making it impossible to use them as an attribute.

[lrafeei]

Oh, I see--In this case, the decision to keep the graphql attributes in the span (and not the transaction to which the span belongs) was made primarily due to the case of batch querying. In that case, multiple queries and/or mutations can be sent within a single transaction and there would be multiples of the same attributes.

As an example, let's say that there is a batch query with one query and one mutation:

[
  {
    query: query GetBookForLibrary {
      library(branch: "downtown") {
        books {
          title
          author {
            name
          }
        }
      }
    }
  },
  {
    query: mutation {
      addThing(name: "added thing!")
    }
  }
]

Here, we have two graphql.operation.type, two graphql.operation.name, et cetera. To avoid that conflict, these attributes are confined to the operation span or the resolver span.

As for your specific issue regarding the overriding of the graphql.operation.name attribute, I am not sure of a good way to override the name of the span, but perhaps you can utilize the newrelic.agent.add_custom_span_attribute(key, value) API?