feat(autoresearch): persistent build caches — minimal-mode gatsby build ~477s → ~82s#18661
Conversation
Persistent, content-revalidated caches plus targeted fixes, measured with pnpm clean && GATSBY_MINIMAL=true pnpm build between every change: - relocate webpack's filesystem caches outside .cache so `gatsby clean` stops discarding them: site build-javascript/build-html stages (gatsby-node.ts), gatsby core's query-engine and page-ssr engine bundlers (patches/gatsby+4.25.9.patch), and gatsby-plugin-mdx's feed html renderer (patch, incl. the missing compiler.close() without which webpack 5 never flushes that cache to disk). Webpack revalidates every entry against module contents and build dependencies, so stale entries miss instead of misresolving. - persist gatsby-plugin-mdx genMDX payloads (headings/body/html source data) outside .cache, keyed by node contentDigest plus a fingerprint of compile-affecting config (plugin version, each gatsby remark plugin's entry-file hash and options) - compile the feed html renderer once per process instead of once per DataLoader batch (was dozens of identical near-full-app webpack compiles during onPostBuild) - trim the sitemap's community-question fetch to fields=[permalink] (was populate=* — ~50x the transfer for identical output) - disable gatsby telemetry in build scripts and enable Node 22's NODE_COMPILE_CACHE for the build process tree - wire the persistent caches into CI following the existing warm-on-master/restore-on-PR pattern (cache-warmup.yml saves daily, deploy-preview.yml restores keyed on lockfile+patches) Local steady state: 476.8s -> 82.4s (phases: webpack trio 255s -> 18s, createPages 57s -> 1.5s, onPostBuild 67s -> 13s). First build after a dependency, patch, or config change re-primes the invalidated caches and runs slower; steady state resumes on the next build. Generated-By: PostHog Code Task-Id: 0cdc3a81-9997-4d76-91d3-03e2b599033c
Deploy preview
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ef3d443ce5
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Bundle reportTotal JS (gzip)6.73 MiB (-5.9 KiB / -0.1%) Largest changed named chunks
Eager graph (modules shipped in each entrypoint's initial chunks)
Largest modules in the
|
| Module | Size |
|---|---|
./src/data/mcp-tools.json |
839.7 KiB |
css ./node_modules/.pnpm/css-loader@5.2.7_webpack@5.101.3/node_modules/css-loader/dist/cjs.js??ruleSet[1].rules[8].oneOf[1].use[1]!./node_modules/.pnpm/postcss-loader@4.3.0_postcss@8.5.6_webpack@5.101.3/node_modules/postcss-loader/dist/cjs.js??ruleSet[1].rules[8].oneOf[1].use[2]!./src/styles/global.css |
738.5 KiB |
./src/components/Stickers/Stickers.tsx |
696.4 KiB |
./node_modules/.pnpm/@radix-ui+react-icons@1.3.2_react@18.3.1/node_modules/@radix-ui/react-icons/dist/react-icons.esm.js |
481.4 KiB |
./node_modules/.pnpm/rehype-raw@7.0.0/node_modules/rehype-raw/lib/index.js + 29 modules |
395.1 KiB |
./node_modules/.pnpm/@posthog+icons@0.36.6_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.cjs.js |
364.8 KiB |
./src/hooks/useCustomers.tsx + 54 modules |
355.1 KiB |
./node_modules/.pnpm/@posthog+icons@0.36.6_react-dom@18.3.1_react@18.3.1__react@18.3.1/node_modules/@posthog/icons/dist/posthog-icons.es.js |
354.8 KiB |
./node_modules/.pnpm/react-markdown@8.0.7_@types+react@16.14.66_react@18.3.1/node_modules/react-markdown/lib/react-markdown.js + 88 modules |
351.4 KiB |
./node_modules/.pnpm/cloudinary-core@2.14.0_lodash@4.17.21/node_modules/cloudinary-core/cloudinary-core.js |
281.9 KiB |
./src/components/ProductComparisonTable/index.tsx + 120 modules |
278.6 KiB |
./src/components/SearchUI/index.tsx + 87 modules |
271.8 KiB |
./gatsby-browser.tsx + 45 modules |
248.0 KiB |
./node_modules/.pnpm/d3@7.9.0/node_modules/d3/src/index.js + 208 modules |
247.4 KiB |
./src/components/Pricing/PricingSlider/Slider.tsx + 87 modules |
239.9 KiB |
Eager-graph budgets are report-only until a baseline is established. Sizes are gzip of public/**/*.js; eager size is webpack module source bytes for the modules actually shipped in the entrypoint's initial chunks (post-tree-shake).
- Fingerprint gatsby remark plugins by their runtime pluginOptions (the
field get-source-plugins-as-remark-plugins.js actually passes to each
plugin) falling back to the raw config options, and hash the resolved
modulePath when present. Previously an options-only change (e.g.
gatsby-remark-autolink-headers { icon }) would not rotate the
fingerprint, letting restored payload caches serve stale body/html.
Bumped the fingerprint version to invalidate previously saved entries.
- Preserve gatsby's stage buildDependencies (its webpack config plus
every plugin's gatsby-node closure) in the mdx html renderer's cache
config instead of replacing them with just the patched file, so site
webpack config changes (aliases/loaders) still invalidate the
renderer's persistent cache across restored builds.
Generated-By: PostHog Code
Task-Id: 0cdc3a81-9997-4d76-91d3-03e2b599033c
|
Note 🤖 Bot comment — written by PostHog Code (AI agent) on Paul's behalf. Both Codex findings verified against the code and fixed in f09e655:
Verified the updated patch applies cleanly from the pristine tarball with all modification markers present. |
|
|
Note 🤖 Bot comment — written by PostHog Code (AI agent) on Paul's behalf. Can the two patches in this PR be retired by dependency updates?No — checked against the registry and upstream source rather than assumed:
|
1 participant
Created with Autoresearch.
Stacked on #18650 (sourcing-phase optimizations). This run targeted the whole
GATSBY_MINIMAL=true pnpm buildwall time — the command the CI preview job runs — measured locally withpnpm cleanbetween every change.Results (local, quiet host, steady state)
.cache/genMDXpayloads (config-fingerprinted)compiler.close())fields=[permalink]instead ofpopulate=*GATSBY_TELEMETRY_DISABLED=1in build scriptsNODE_COMPILE_CACHEfor the build process treePhase-level: webpack trio 255s → 18s, createPages 57s → 1.5s, onPostBuild 67s → 13s, run-queries 17s → 2.5s.
Why output is unchanged
gatsby cleanstops deleting it. The repo's CI comment about.cacherestores causing MDX incoherence doesn't apply: that was Gatsby's data-layer cache, not webpack's content-keyed packs.contentDigestplus a fingerprint of everything that changes compile output (plugin version, each gatsby remark plugin's entry-file sha1 and options) — bypassing gatsby's wipe-on-config-change means the key must carry the config, and it does. Verified headings/anchors render identically.renderMdxBodyat runtime); one compile per process renders byte-identical html to one per batch. Thecompiler.close()addition matches gatsby core's own engine bundlers and is required for webpack 5 to flush its filesystem cache at all.attributes.permalink; verified the trimmed query returns identical permalinks (76 pages, 7565 questions).NODE_COMPILE_CACHEis Node 22's supported V8 bytecode cache, content-validated.Verified after every measured change: 2009 HTML pages, RSS feed 65 items, spot-checked docs heading anchors.
Operational notes
package.jsonvia browserslist) and runs at roughly pre-cache speed; steady state resumes on the next build. This bit us twice during measurement (iterations 4 and 7) before we learned to prime first.node_modules/.cache/locally; the CI artifact (saved daily bycache-warmup.yml, restored bydeploy-preview.yml, keyed on lockfile+patches with prefix fallback) compresses to well under GitHub's 10GB repo quota.pnpm clean'd runs with per-phase timer attribution.Created with PostHog Code