benchmark: test performance cost of re-creating schema

[IvanGoncharov]

Nessary as part of a current and possible future disscussion on perfomance penalties of schema
transformation.

[netlify]

✅ Deploy Preview for compassionate-pike-271cb3 ready!

Name	Link
🔨 Latest commit	1f695da
🔍 Latest deploy log	https://app.netlify.com/sites/compassionate-pike-271cb3/deploys/628556043441380008ef4efa
😎 Deploy Preview	https://deploy-preview-3596--compassionate-pike-271cb3.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[github-actions]

Hi @IvanGoncharov, I'm @github-actions bot happy to help you with this PR 👋

Supported commands

Please post this commands in separate comments and only one per comment:

• @github-actions run-benchmark - Run benchmark comparing base and merge commits for this PR
• @github-actions publish-pr-on-npm - Build package from this PR and publish it on NPM

[github-actions]

@github-actions run-benchmark

@yaacovCR

Benchmark output

graphql@16.4.0 benchmark
ts-node resources/benchmark.ts "--revs" "HEAD" "BASE"

🍳 Preparing HEAD...
🍳 Preparing BASE...
⏱ Recreate a GraphQLSchema
�[36m1�[0m tests completed.
�[36m2�[0m tests completed.

HEAD�[90m x �[0m�[32m454�[0m ops/sec �[90m±�[0m�[33m2.35�[0m�[36m%�[0m�[90m x �[0m121 KB/op�[90m (16 runs sampled)�[0m
�[32mBASE�[0m�[90m x �[0m�[32m457�[0m ops/sec �[90m±�[0m�[32m0.97�[0m�[36m%�[0m�[90m x �[0m129 KB/op�[90m (16 runs sampled)�[0m

⏱ Build Schema from AST
�[36m1�[0m tests completed.
�[36m2�[0m tests completed.

HEAD�[90m x �[0m�[32m29.46�[0m ops/sec �[90m±�[0m�[32m1.70�[0m�[36m%�[0m�[90m x �[0m1.67 MB/op�[90m (6 runs sampled)�[0m
�[32mBASE�[0m�[90m x �[0m�[32m29.68�[0m ops/sec �[90m±�[0m�[32m1.67�[0m�[36m%�[0m�[90m x �[0m1.67 MB/op�[90m (6 runs sampled)�[0m

⏱ Build Schema from Introspection
�[36m1�[0m tests completed.
�[36m2�[0m tests completed.

�[32mHEAD�[0m�[90m x �[0m�[32m34.82�[0m ops/sec �[90m±�[0m�[32m0.37�[0m�[36m%�[0m�[90m x �[0m1.78 MB/op�[90m (7 runs sampled)�[0m
BASE�[90m x �[0m�[32m33.88�[0m ops/sec �[90m±�[0m�[31m5.97�[0m�[36m%�[0m�[90m x �[0m1.78 MB/op�[90m (7 runs sampled)�[0m

⏱ Execute Introspection Query
�[36m1�[0m tests completed.
�[36m2�[0m tests completed.

HEAD�[90m x �[0m�[32m 28.4�[0m ops/sec �[90m±�[0m�[32m1.28�[0m�[36m%�[0m�[90m x �[0m586 KB/op�[90m (5 runs sampled)�[0m
�[32mBASE�[0m�[90m x �[0m�[32m28.68�[0m ops/sec �[90m±�[0m�[32m0.76�[0m�[36m%�[0m�[90m x �[0m586 KB/op�[90m (5 runs sampled)�[0m

⏱ Parse introspection query
�[36m1�[0m tests completed.
�[36m2�[0m tests completed.

�[32mHEAD�[0m�[90m x �[0m�[32m10,501�[0m ops/sec �[90m±�[0m�[32m0.09�[0m�[36m%�[0m�[90m x �[0m1.41 KB/op�[90m (26 runs sampled)�[0m
BASE�[90m x �[0m�[32m10,468�[0m ops/sec �[90m±�[0m�[32m0.29�[0m�[36m%�[0m�[90m x �[0m1.33 KB/op�[90m (26 runs sampled)�[0m

⏱ Validate Introspection Query
�[36m1�[0m tests completed.
�[36m2�[0m tests completed.

�[32mHEAD�[0m�[90m x �[0m�[32m482�[0m ops/sec �[90m±�[0m�[32m0.30�[0m�[36m%�[0m�[90m x �[0m86.63 KB/op�[90m (10 runs sampled)�[0m
BASE�[90m x �[0m�[32m482�[0m ops/sec �[90m±�[0m�[32m0.34�[0m�[36m%�[0m�[90m x �[0m86.68 KB/op�[90m (10 runs sampled)�[0m

⏱ Validate Invalid Query
�[36m1�[0m tests completed.
�[36m2�[0m tests completed.

�[32mHEAD�[0m�[90m x �[0m�[32m395�[0m ops/sec �[90m±�[0m�[32m0.24�[0m�[36m%�[0m�[90m x �[0m160 KB/op�[90m (9 runs sampled)�[0m
BASE�[90m x �[0m�[32m394�[0m ops/sec �[90m±�[0m�[32m0.92�[0m�[36m%�[0m�[90m x �[0m160 KB/op�[90m (9 runs sampled)�[0m

⏱ Validate SDL Document
�[36m1�[0m tests completed.
�[36m2�[0m tests completed.

�[32mHEAD�[0m�[90m x �[0m�[32m44.59�[0m ops/sec �[90m±�[0m�[32m0.58�[0m�[36m%�[0m�[90m x �[0m346 KB/op�[90m (7 runs sampled)�[0m
BASE�[90m x �[0m�[32m44.53�[0m ops/sec �[90m±�[0m�[32m0.60�[0m�[36m%�[0m�[90m x �[0m348 KB/op�[90m (7 runs sampled)�[0m

⏱ Visit all AST nodes
�[36m1�[0m tests completed.
�[36m2�[0m tests completed.

�[32mHEAD�[0m�[90m x �[0m�[32m190�[0m ops/sec �[90m±�[0m�[32m0.23�[0m�[36m%�[0m�[90m x �[0m288 KB/op�[90m (14 runs sampled)�[0m
BASE�[90m x �[0m�[32m190�[0m ops/sec �[90m±�[0m�[32m0.29�[0m�[36m%�[0m�[90m x �[0m288 KB/op�[90m (14 runs sampled)�[0m

⏱ Visit all AST nodes in parallel
�[36m1�[0m tests completed.
�[36m2�[0m tests completed.

�[32mHEAD�[0m�[90m x �[0m�[32m18.25�[0m ops/sec �[90m±�[0m�[32m0.20�[0m�[36m%�[0m�[90m x �[0m437 KB/op�[90m (5 runs sampled)�[0m
BASE�[90m x �[0m�[32m18.24�[0m ops/sec �[90m±�[0m�[32m0.21�[0m�[36m%�[0m�[90m x �[0m437 KB/op�[90m (5 runs sampled)�[0m

[yaacovCR]

Hmmm. Am I write in reading this as about 450 ops/s = about 2.2 ms to recreate a schema? One note is that this is additive if a pipeline does this repeatedly.

Note that at the last WG, we discussed not just simple schema recreation, but also potentially "rewiring" it, i.e. recreating all types that reference changed types. In graphql-tools, mapSchema changes only the requested types, and then calls rewireTypes which unfortunately recreates ALL schema types. This ensures that all fields on all types are pointing to the new types, but it may often be overkill.

Within graphql-js, the analogue would be lexicographicallySortSchema which does the same thing, I believe.

[IvanGoncharov]

Point of this benchmark is to prove that you can attach configuration to GraphQLSchema and force users to re-create schema if they want to change config.
So it's related to `getFieldDef' discussion, discussion about new flag to disable introspection.
And few other related proposal I have, it's not directly related to assigning resolvers.

Within graphql-js, the analogue would be lexicographicallySortSchema which does the same thing, I believe.

lexicographicallySortSchema doing sorting which computation intensive on itself and scales non-linear with number of items so comporation with simply assigning properties is not fair.
Also it sort bunch of things like "input fields", args, etc. that don't have resolvers.
Even under this condition lexicographicallySortSchema takes 10ms on my machine (probably 20ms on GitHub's worker).
So I think properly implemented resolver attaching function will take 5ms tops on a huge schema.

Hmmm. Am I write in reading this as about 450 ops/s = about 2.2 ms to recreate a schema? One note is that this is additive if a pipeline does this repeatedly.

Yes, it's additive but you can use caching (ES6 Map of transformation) if you want to do it per request.
Also note it's worse case performance since it run on default configuration of GitHub CI inside docker (probably inside VM). On my machine it's stable 970 ops/s which is ±1 ms.

Also please note it's done for absolutely massive GitHub schema which is way way bigger that most users will ever have.

[yaacovCR]

We are optimizing for cold starts for lambda-like environments where caching is not possible.

[IvanGoncharov]

We are optimizing for cold starts for lambda-like environments where caching is not possible.

So if it's a per-request transformation (e.g. different schema based on headers, URL, etc.) when it can be cached.

If transformation is done before starting the server, I think 2.2ms on a weak CI runner for GitHub size schema is not a big price to pay.

[yaacovCR]

I guess it's important to keep in mind the baseline cost of a cold start for the execution environment to spin up.

https://mikhail.io/serverless/coldstarts/aws/

If it's usually 400 ms, then an extra 2 ms could represent some huge increase with respect to the user code, but still less than a 0.5% increase compared to the overall startup delay, not so important of an optimization after all!

[yaacovCR]

Whoops. Other solutions measure cold starts in ms time. https://blog.cloudflare.com/eliminating-cold-starts-with-cloudflare-workers/ Sorry for the multiple comments.