twitter + graphql-java

[jbellenger]

Hi team! Twitter is wrapping up its migration to graphql-java and I wanted to share some info about how we use graphql and why we chose to migrate our system to graphql-java.

Some background on how we use graphql: we use one unified schema to serve data to first-party twitter clients and to power our rest api. The schema defines the entirety of our api data model and includes roughly 1000 object types, 100 input types, and 300 mutation fields. The schema itself grows daily as several hundred developers across the company add fields and types to the schema to support their projects.

Our graphql api serves around 500k requests per second, and the nature of our api is that some of our largest and most complex queries have the highest request rate. For example, one of our most frequently-executed queries is 2500 lines long and returns 50k fields per request. That's a lot of fields!

While we've been operating this graphql system for a couple of years, we encountered some performance issues last spring when we regularly started serving requests that returned 20k+ fields. We found that our graphql implementation was instantiating a large number of intermediate objects during execution, creating significant memory pressure and destabilizing our service when under heavy load.

For comparison we ran some execution benchmarks against other graphql implementations and were surprised at graphql-java's performance properties. For some of our most complex queries, graphql-java had 3x the throughput of our current graphql implementation, as well as predictable memory performance that made it stable under high load, allowing us to make more efficient use of our hardware.

It took 8 engineers 6 months to port our graphql schema generator and execution system to graphql-java. As of last week, we're serving all of twitter's graphql query operations on graphql-java and we're already seeing great results, including a 35% reduction in overall cpu utiization and a 25% reduction in GC time. And these gains are within a totally untuned system; we expect to realize more gains after we finish our migration and can focus on fine tuning.

Our team is excited about the performance improvements, but we are even more excited about participating in the healthy developer community that surrounds this project. We look forward to contributing patches back to graphql-java and finding other ways to be useful.

Much love,
The Twitter Core API Platform team
@arshij @chriswr95 @folone @haggaikaunda @jbellenger @schenkman

[sbilello]

@jbellenger
Did you try to leverage the https://github.com/Netflix/dgs-framework built on top of graphql-java?

[jbellenger]

Did you try to leverage the https://github.com/Netflix/dgs-framework

We didn't (though we've met with the netflix folks a couple of times and are big fans of their work).

One of the quirks of our graphql system is that our developers don't define graphql objects directly. Instead, data is described using an abstract type system that our graphql system then compiles into a schema. So a lot of what dgs provides is not relevant to the primary job of our system. Mostly what we want is a solid and high-performance implementation of graphql that keeps pace with the graphql spec, which graphql-java provides.

[andimarek]

hey @jbellenger ... thanks a lot for this public and very positive feedback.

very happy to have Twitter onboard 😄

[bbakerman]

and returns 50k fields per request.

I would love to get more insight on the business case for 50,000 bits of data being needed in one call.

graphql systems often do best when you ask for what you need and no more - and hence avoid under fetching and over fetching.

Can you explain why 50K bits of data is useful? (not saying its not useful - just trying to get insight into this)

[jbellenger]

I agree that 50k is a lot! This number reflects a perfect storm of factors:

• some of our clients were originally built around monolithic data objects and aren't at the point where they can request granular bits of data. Getting them to that point is a work in progress but will take a while.
• this particular request retrieves all the data for a product surface that composes a large number of entities and is also a hot spot for product development, so it's always going to be on the high end of field counts.
• internally, we're still actively focused on improving the reach and scope of our graphql api. This is a higher priority for us than query optimization.

At a high-level, I think all of these points are really artifacts of how we migrated from rest to graphql rather than starting fresh. I'd be interested in hearing other companies development stories of how they adopted graphql.

[soneymathew]

@jbellenger Does your team use persisted queries?

[jbellenger]

Not only do we use persisted queries, we require them in production. There are some commonly touted performance benefits of persisted queries, but one of the main benefits we've seen is that it is quite useful to have a document id that can be associated with metrics, privacy reviews, and other internal processes.

[davidafsilva]

Hey @jbellenger,

Thank you for the insight!

With those complex queries I was just wondering what latencies are you guys experiencing on average and worst cases (p99.+). Can you share those details?

Mostly curiosity since they're somewhat subjetive due to a number of factors.

[jbellenger]

Execution latency depends on a number of factors which make them difficult to compare. For example, our end-to-end latency is dominated by our data fetcher latency which is out of our hands. Other complications include parallelism of queries, instrumentation of the executor, and configuration of the hardware and jvm.

Instead of comparing latency, we've found that it's been more useful to compare the maximum serving throughput for real requests when running on the same hardware configuration.

To offer a data point, one of our recent tests showed that for a 25k fields/request operation, on one of our beefier-than-average hardware configurations, our GJ-based system could serve 150 rps compared to 60 rps of our incumbent system.

[kfwerf]

Hi @jbellenger

Thanks for the intro and cool to read some of the Twitter insights. What was the reason to choosing for this setup over a federated setup?

It sounds like you created a unique system on top of graphql-java, do you have any plans to open source this? E.g. graphql schema generator and execution system.

[jbellenger]

We haven't seen a need to federate our graphql schema because our underlying data layer (known as strato) is itself federated. Having federation as a data-layer building block gives us most of the perks of federation while also allowing us to have a unified schema with strong network effects and without too many performance gotchas.

There aren't any current plans to open source either our schema generator or strato, though there have been a decent number of talks describing how these components work together. Some background talks that (IMO) are interesting are:

• Introducing Stitch
• Strato: Twitter’s Virtual Database Powered by Microservices

[azarichnyi]

Hi @jbellenger

Thank you for a very interesting discussion（っ＾▿＾）<3

How do you store persisted queries? Do you have only pre-registered queries or do allow clients to register new queries if queries are unknown to the server?

[jbellenger]

Hi @azarichnyi !

Our handling of persisted queries is a little different from APQ. At a high-level, most of our clients make api requests using a documentId+operationName pair and do not include the text of the graphql query:
GET /graphql/123abc/GetTweet?variables=tweet_id=20

The document here, 123abc, is the id of a pre-registered query. Our clients (of which there are fewer than 10) have graphql build system integrations that compile+validate+store graphql documents that are used by the application. This build step provides a documentId to the application that can be used in place of a graphql query text.

For requests that originate outside of the twitter data center, we only allow pre-registered queries. This is a useful rule because it prevents untrusted clients from introspecting our schema, potentially discovering unannounced features or constructing pathologically expensive requests. Having a documentId for all requests is also useful when it comes to metrics, and allows us to
track field counts, errors, latencies, and other metrics on a per-document and per-operation basis.

[jord1e]

In this context, what is the difference between a document(id) and operationName?

Is the "document" a set/list of pre-build operations? (do you enforce operations having a operation name?)

Is the document id immutable, or is it different when you add/remove operations to/from it?

This is quite interesting from an architectural standpoint, thanks for the insights

[jbellenger]

Hi @jord1e !
We mean document to be what's described by the spec, which is basically a graphql text that defines 1 or more graphql operations.

Our documentIds are generated from a hash of the "normalized" document. We do this by parsing the provided document, removing comments and whitespace, re-rendering that document to a string, and then generating a hash value from that string. The effect of this is that changes to a document that impact its runtime execution will produce a new documentId when stored, but changes that don't change its execution will not change the documentId. This is useful for identifying documentId-specific performance issues, as we can be confident that all documents with the same documentId have the same performance characteristics.

An operationName is the name of an operation within the document. The graphql spec allows documents to define multiple operations, though in practice none of our clients use this. The graphql spec also allows documents that define only one operation to not have to name that operation -- we require that all our operations be named, even in small documents. The effect of this is that for a given operation, like GetTweet, we can track the performance of that operation over time as the documents that it is defined in evolve.

Re: operation name consistency: it's something of a delicate balance. Having different clients use the same operation name for similar operations can simplify some internal processes like privacy reviews and capacity planning, though making that a hard requirement takes some power away from our developers, which is somewhat counter to why they're using graphql in the first place. I'd be curious to see how other deployments of graphql approach this.

[CEikermann]

@jbellenger as far as I know Twitter is developing using scala, does it means you have developed a thin scala wrapper around of graphql-java? If yes, do you think it would be possible to open source this?

[jbellenger]

Hi @CEikermann !
You're right that most services at Twitter are written in scala, including the graphql api that uses graphql-java.

The nature of scala is that it has native interop with java libraries, allowing scala programs to directly use java-authored libraries like log4j, jackson, graphql-java etc. We have no plans to open-source the libraries that generate our graphql schema or call into graphql-java for execution. I promise you that it is a large amount of proprietary boilerplate that is not interesting or applicable outside of Twitter.

We do expect to contribute changes to graphql-java that improve features and performance that will be useful to users of graphql-java. We think this is the best way to serve the community that uses this library and graphql at large.

[CEikermann]

Thanks for your fast reply!

[bbakerman]

A little data point on how Twitter has already helped the graphql-java community

We just did some HotSpot analysis of our Atlassian Graphql Gateway and we found that for certain large queries the RulesVisior code was a hotspot of object allocation and also time spent.

7.6% - 29,265 ms java.util.LinkedHashSet.<init>(java.util.Collection)
  5.5% - 21,260 ms graphql.validation.RulesVisitor.enter


public class RulesVisitor implements DocumentVisitor {

    public void enter(Node node, List<Node> ancestors) {
        validationContext.getTraversalContext().enter(node, ancestors);
        Set<AbstractRule> tmpRulesSet = new LinkedHashSet<>(this.rules);
        tmpRulesSet.removeAll(rulesToSkip);
        List<AbstractRule> rulesToConsider = new ArrayList<>(tmpRulesSet);
        if (node instanceof Document){
        

and the we remembered ... didn't Twiitter fix that?

#2563

We upgraded internally to a daily version and... hey presto... this fell out of our hotspots!

Thanks Twitter

[folone]

Nice! Can we expect a new release with this change in soon? 🙏