full page caching for rustdoc pages in the CDN

[syphar]

We can use a CDN to improve world-wide speed for serving documentation. Even when we would improve server-side response times (for example by caching S3 requests locally on the webserver) we would look at the global latency between EU/US and else where (at least 100ms). If after CDN caching we still need to optimize server-side response times we can do it then.

Documentation is mostly static and can only change with a build it's a nearly perfect candidate for CDN caching. It's only nearly because we have rebuilds and also we are adding a header & footer.

I think we can leverage CDN caching for most parts of the site. When actively invalidating caches and using a good CDN we would still always have up-to-date content.

page-types and invalidation events

cached forever, no invalidation needed:

• static assets with hashed filenames

can only change after any release for one specific crate

• rustdoc pages (the header contains all versions of the crate)
• latest-version redirects
• release-internal redirects

can only change when we release new code:

• documentation pages etc
• styles

not really cachable:

• search-results
• release-lists (only cachable if we accept them being outdated for a certain amount of time)

requirements to the CDN

we need

• fast invalidation (tag-based if possible, path/pattern based works for simple cases too).
• CDN specific caching headers will be removed on the CDN level (to have control over the cache at all times)
• logic on the edge to add CSP nonces at the edge

Nice to have would be:

• serving stale content while updating the cache in the background.
• soft purge: serves stale content while updating the cache. Prevents thundering herd problem when clearing the cache.

CloudFront

• invalidations are probably too expensive to execute these on every release
• path/pattern invalidations are possible, tags not
• invalidations take minutes, sometimes 15.
• secret headers I would need to research, I don't have a definitive answer yet. Perhaps solvable with lambda@edge or CF configuration.
• but we already have it
• Lambda@edge could probably solve the CSP issue, I didn't dig deeper on programming language support in there

Fastly

• invalidations are free, and take 100ms worldwide
• tag-based invalidations work,
• secret headers too
• soft purge and serving stale content is natively supported
• fastly compute@edge has rust-support we could use for CSP (POC : https://github.com/syphar/docs-rs-fastly-csp/blob/main/src/main.rs )
• PyPI gets fastly for free, I guess we could get it too.
• but we don't have it yet, SSL needs work, contracts too.

CloudFlare

I didn't dig deeper yet on the feature set here.

browser caches

since we want to actively invalidate certain caches we won't cache these pages in the browser and limit browser caching to static assets with hashed filenames as currently.

[syphar]

@jyn514 to start with I just wrote down my thoughts. I'll probably refine the text over the time and add details.

The quickest solution with the least amount of work would be:

• use our current CloudFront setup
• cache pages and redirects below the /crate_name/ path, invalidate this whole pattern on every release (or yank) for that crate
• release-lists and static pages could be cached for a short time (30 minutes or shorter, or not at all)
  a full cache purge is risky depending on the amount of requests the server gets because all cache locations would request new content at the same time. costs need to be checked -> AWS charges $0.005 per path invalidated after the first 1000 invalidations (standard price, I don't know our discounts / credits). When I take 600 releases per day, that sums up to $90, which feels ok-ish, but not up to me :)

The best solution would probably be going with fastly (alternatively cloudflare if the features match).
From a backend perspective the effort is similar (also I did some setups already). Additional work is of course the contract / infrastructure parts.

[syphar]

to add: for CloudFront there would be additional costs for Lambda@Edge

[jsha]

Thanks for writing this up! Could you describe what is the current status quo? CloudFront, but with no caching? Or CloudFront, but without lots of PoPs all over the world?

It would be great to have a summary of how bad the current situation is, and how much it would improve under a better CDN approach. Are you interested in collecting a bunch of samples from https://webpagetest.org/? It seems like in particular Time To First Byte (TTFB) would be the most important measure that would be improved by a better CDN.

Also cross-linking some caching/performance related issues:

• big crates are slow to use with new archive-storage #1528
• Make it possible to use docs.rs offline for pages that have been visited at least once #845
• Downloadable docs #174

Another thing to consider: With Cloudflare Workers / Fastly Compute@Edge, we could do the unpacking of storage blobs inside the CDN. That would have the advantage that when someone requests 1 page of a crate's docs, their local PoP would have the whole blob of that crate's docs, so subsequent navigations would be very fast.

AWS charges $0.005 per path invalidated after the first 1000 invalidations (standard price, I don't know our discounts / credits). When I take 600 releases per day, that sums up to $90, which feels ok-ish, but not up to me :)

I'm getting 0.005 * 600 = $3. Presumably there is some other multiplier here that I'm missing?

[syphar]

Thanks for writing this up! Could you describe what is the current status quo? CloudFront, but with no caching? Or CloudFront, but without lots of PoPs all over the world?

Valid question, thanks for asking.

The whole of docs.rs is behind CloudFront. All static assets, from rustdoc or docs.rs, are cached in the browser and the CDN. All other pages are uncached and just routed through the CDN. Our webserver answers all of them. For most crates server-side response times are totally fine. From my perception the bottleneck is the request from europe (for me) to the AWS datacenter in the US where docs.rs is hosted.

Right now the pages are regenerated for every request, including fetching the original files from S3. While we could of course start caching files locally on our webserver we could just skip this step and directly cache on the edge, helping not only US but worldwide users :)

It would be great to have a summary of how bad the current situation is, and how much it would improve under a better CDN approach. Are you interested in collecting a bunch of samples from https://webpagetest.org/? It seems like in particular Time To First Byte (TTFB) would be the most important measure that would be improved by a better CDN.

Any modern CDN would be fine regarding performance and POPs. Biggest differentiator is how we can selectively invalidate parts of the page, since every new (or re-) release would also change cached content for old releases and we want the docs to be up-to-date.
Edge-Logic is (for the start) only needed to combine CSP with caching.

TTFB measurements could of course be fed into a better CDN selection, we could start with CloudFront and already have a good solution without too much infrastructure effort.

Another thing to consider: With Cloudflare Workers / Fastly Compute@Edge, we could do the unpacking of storage blobs inside the CDN. That would have the advantage that when someone requests 1 page of a crate's docs, their local PoP would have the whole blob of that crate's docs, so subsequent navigations would be very fast.

While that could be a next optimization step, it needs more design, since the documentation blobs sometimes have multiple gigabytes and millions of files.

AWS charges $0.005 per path invalidated after the first 1000 invalidations (standard price, I don't know our discounts / credits). When I take 600 releases per day, that sums up to $90, which feels ok-ish, but not up to me :)

I'm getting 0.005 * 600 = $3. Presumably there is some other multiplier here that I'm missing?

Sorry I wasn't clear enough here, I was thinking about $90 monthly.

[jsha]

I think this sounds exactly right. I was actually wondering if something like this (CDN-only caching for the HTML pages) was possible with CloudFront.

[syphar]

@jsha i actually removed the long writeup again because I forgot the main reason why full page caching is more work here: CSP 😄.

We could handle this part with lambda@edge, but I would have to dig into that first.