docs deploy: production builds are OOM-failing now — the peak is the Turbopack compile phase, not the prerender; turbopackSourceMaps is a free knob #12683's option set missed #12711
Description
Activity
Claimed. session
3100934e-af9d-4104-9145-a37b8280a504· branchclaude/issue-12711-docs-build-sourcemaps· worktreesort-dot-path-fallback-fc72e9.Scope for this claim: the
experimental.turbopackSourceMaps: falsechange toapps/docs/next.config.mjsplus its changeset — nothing else. The build-machine decision (#12683 option A) stays with the maintainer and is untouched by this PR; it remains available if the measured headroom turns out insufficient on Linux.- added a commit that references this issue
on Aug 27, 2026 Result of the merged fix: the failure mode changed, the outage did not end
PR #12712 merged as
18b6d89c. The production build it triggered (dpl_BF8NovndAJa5ezngzUJPphn41hXB) failed, but not the way its predecessors did:before the fix after the fix errorCodeout_of_memoryBUILD_EXCEEDED_MAXIMUM_TIMEhow it ended SIGKILL at 2m / 11m23s ran the full 46.1 min and timed out The flag is confirmed applied on Vercel — the build log prints
⨯ turbopackSourceMapsalongside· cpus: 2, byte-identical to the local run where source maps measurably went 260 files/348 MB → 0 and peak RSS dropped 434 MB.Reading
This is the signature of a process that stopped crossing the OOM line but is still pressed against it. It entered
Creating an optimized production build ...at 13:28:53 and emitted nothing for the next 27+ minutes; the same compile takes 20s locally. Reclaim pressure near the container ceiling is the only thing that accounts for a 100x slowdown on work that is not I/O bound, and it is exactly what the earlier 11m23s-of-silence-then-SIGKILL was already showing in weaker form.So the −8.4% was real and it was not enough: it bought just enough headroom to avoid the kill and not enough to let the compiler run. The ~15% projected margin this card flagged as "thin" turned out to be thin in the direction that matters.
What this retires
- All remaining free in-repo knobs.
turbopackScopeHoisting,--max-old-space-size,turbopackFileSystemCacheForBuildmeasured as noise (matrix in the issue body);turbopackMinify: falsewould give another 972 MB but inflates client JS 5.8 MB → 16 MB (+176%) and cannot be confined to the server side;includeProcessedMarkdown: falsebreaksgetLLMText. There is no further free number to lower. - The hypothesis that the fix alone could clear the ceiling. It cannot. Measured, not projected.
What it confirms
Memory is the binding constraint, and the container is undersized for this build — now demonstrated from both sides: below the line it is killed, at the line it is too slow to finish. #12683 option A (a larger build machine) is the remaining route, and this run is the evidence its decision was waiting on. Current machine:
standard, 4 cores / 8192 MB, on aproteam.The merged change stays: it is a net win independently of the machine (348 MB less build output, −434 MB peak, faster compile), and on a larger container it keeps that margin rather than spending it.
⚠️ Note for whoever acts on this: the site is still servingdpl_2nfWjGjSwZjakVEmUG1kBWD6697r, unchanged since before the outage.- All remaining free in-repo knobs.
Resolved — the outage is over, and the fix was the machine
The maintainer moved the docs project to an Enhanced build machine (8 vCPU / 16384 MB, project-level only; the other six projects stay on
standard). Vercel had independently reached the same conclusion, recordingbuildMachineElasticReason: "build-timeout-failure"on the project.Result, measured:
Standard (4 vCPU / 8 GB) Enhanced (8 vCPU / 16 GB) outcome out_of_memory, thenBUILD_EXCEEDED_MAXIMUM_TIMEREADY wall time 46–50 min 3.7 min cost/build 184–200 CPU-min ≈ $0.64–0.70 (wasted) 30 CPU-min ≈ $0.105 https://objectstack.ai/now servesdpl_3YG2asYDmqBzoQt9eiicVLgZYivP— off the pinneddpl_2nfWjGjSwZjakVEmUG1kBWD6697rfor the first time since the outage began. HTTP 200,/docsrenders. The re-check in #12698 is satisfied.A larger machine turned out to be cheaper per build, not more expensive. Same $0.0035/CPU-minute across tiers, and doubling the cores cut wall time by ~13x because the slowness was reclaim pressure, not work. The earlier projection in this thread ($110–164/mo) was pessimistic for the same reason.
What the evidence chain established
- The kill was always in the Turbopack compile phase, never the prerender — which retires the attribution in docs build:
next buildholds ~7.6 GB in ONE process — the docs deploy's next memory ceiling, and no in-repo knob bounds it #12683 and clears OG images / markdown routes as suspects. - Peak scales as
2360 MB + 6.21 MB × pageslocally, which extrapolates to≈3682 MB + 9.7 MB × pageson Vercel and predicts 7591 MB at 403 pages against docs build:next buildholds ~7.6 GB in ONE process — the docs deploy's next memory ceiling, and no in-repo knob bounds it #12683's independently measured 7592 MB. 403 pages was at 92.7% of an 8192 MB container — past the safe water line, which is why it had been fine for weeks and then was not. - Free knobs are exhausted:
turbopackSourceMaps: false(merged, −434 MB, kept — it now banks margin instead of spending it) was the only real one.turbopackScopeHoisting,--max-old-space-size,turbopackFileSystemCacheForBuildall measured as noise (±200 MB).turbopackMinify: falsegives −972 MB but inflates client JS 5.8 → 16 MB. Splitting large MDX files does nothing — cost is per page (6.21 MB), not per byte; truncating the 292.9 KBv17.mdxmoved the peak by +191 MB, i.e. noise. - Turbopack's build-time filesystem cache raises peak memory (4591 → 6488 MB) while cutting compile 20s → 6.1s. It trades the resource we were short of for the one we were not; not usable here, though it may become attractive now that headroom exists.
Capacity, going forward
At
≈3682 MB + 9.7 MB/pageagainst 16384 MB, the 75% water line sits near 870 pages; the site is at 403. That is the number to re-check against, and #12743 (avoiding unnecessary builds) reduces how often the ceiling is approached at all.Closing this card. Residual work is tracked in #12743.
- The kill was always in the Turbopack compile phase, never the prerender — which retires the attribution in docs build:
- added a commit that references this issue
on Aug 27, 2026
Production docs deploys are failing right now — five consecutive production builds OOM-killed. This is the live-outage counterpart to #12683's standing-risk card, and it corrects the attribution in both #12683 and #12698.
Measured (Vercel API + build logs, 2026-08-27)
Vercel's own API on the failed deployment:
The peak is the Turbopack COMPILE phase, not the prerender
Both failures die between
Creating an optimized production build ...andCompiled successfully— with zero output in between:The build never reaches
Collecting page data/Generating static pages. So #12683's attribution — "the docs site prerenders 400+ MDX pages", offered as the reason one process holds 7.6 GB — cannot be the mechanism: that phase does not execute. Same for the 403 build-time OG PNGs (app/og/docs/[...slug]/route.tsx) and the 403 markdown routes (app/llms.mdx/docs/[[...slug]]/route.ts) — both aregenerateStaticParamswork in a phase the build dies before.Two of #12683's negative findings are confirmed by measurement below:
--max-old-space-sizeandexperimental.cpusdo not bound this peak.experimental.cpusadditionally cannot matter here for a structural reason — its static-generation workers have not spawned yet when the kill lands.Experiment matrix (local cold builds, 403 MDX, peak single-process RSS)
experimental.turbopackSourceMaps: falseturbopackScopeHoisting: falseNODE_OPTIONS=--max-old-space-size=2048turbopackFileSystemCacheForBuild: trueturbopackMinify: falseincludeProcessedMarkdown: falsegetLLMText//llms.txtdepend on it)Why
turbopackMinify: falseis rejected: client JS goes 5.8 MB -> 16 MB (+176%) — that is real user download weight. Restricting it to the server side (where the memory actually goes: 589 MB of server chunks vs 5.8 MB client) is not available:experimental.serverMinificationis consumed only bydist/build/webpack-config.jsand is never read on the Turbopack path.Why sourcemaps are free here
The build emits 260 sourcemap files totalling 348 MB into
.next/server. Production serverless functions have no use for them. Next's documented build-time default forturbopackSourceMapsfollowsproductionBrowserSourceMaps(false), but the server-side maps are emitted regardless — setting the flag explicitly is what suppresses them.Side benefits beyond the memory: 348 MB smaller build output, and compile time drops 22.6s -> 19.5s locally.
Proposed change
Honest limitation: local measurement is macOS/arm64; Vercel is Linux/x86_64, and the absolute figures differ substantially (local 4.7 GB vs the 7.6 GB #12683 measured). The ratio is the transferable part, and even it is not a guarantee. Scaling -8.4% onto 7592 MB gives ~6950 MB against an 8192 MB container — roughly 15% headroom. That is thin. It is worth trying first because the cost of being wrong is one build cycle, and #12683's option A (bigger build machine) stays available and unaffected.
Also worth noting: Vercel takes 11 minutes to compile what takes 20s locally. A 4-vs-10 core gap does not explain 30x; the rest is thrashing near the container ceiling. Lowering the peak should cut build time substantially as a side effect.
Correction to #12698
#12698's action 2 ("fix the Ignored Build Step — it skips dependency-only changes") rests on a misreading. The project's actual rule is:
exit 1= build,exit 0= skip. So production always builds and Preview always skips — it is not a path-scoped rule. TheCanceled by Ignored Build Stepthat #12698 read off PR #12684's head was that PR's Preview deployment being skipped, which is the intended behaviour. Pushes tomaindo trigger production builds; they have simply been OOM-failing. No dashboard change is needed for the ignore step.Re-check
Refs: #12683 (standing-risk card — this adds an option E its set did not contain) · #12698 (ignore-step correction) · #12677 · #12333 · epic #12243.