test: add adoption skill evals - #97759
Merged
Merged
Conversation
Contributor
Stats from current PR🟢 1 improvement
📊 All Metrics📖 Metrics GlossaryDev Server Metrics:
Build Metrics:
Change Thresholds:
⚡ Dev Server
📦 Dev Server (Webpack) (Legacy)📦 Dev Server (Webpack)
⚡ Production Builds
📦 Production Builds (Webpack) (Legacy)📦 Production Builds (Webpack)
📦 Bundle SizesBundle Sizes⚡ TurbopackClient Main Bundles
Server Middleware
Build DetailsBuild Manifests
Build Cache
📦 WebpackClient Main Bundles
Polyfills
Pages
Server Edge SSR
Middleware
Build DetailsBuild Manifests
Build Cache
🔄 Shared (bundler-independent)Runtimes
📎 Tarball URLCommit: ef35138 |
aurorascharff
marked this pull request as ready for review
August 26, 2026 07:42
…ion-guide' into codex/adoption-skill-evals
…ion-guide' into codex/adoption-skill-evals
This was referenced Aug 27, 2026
aurorascharff
added a commit
that referenced
this pull request
Aug 27, 2026
…ion (#97712) ## What - Documents how legacy `prefetch={true}` behavior changes with Partial Prefetching and how to choose the UI worth preserving. - Updates the adoption skill to capture a passing flag-off `instant()` baseline, use the same assertions to drive the migration, and retain them as regression coverage. - Adds a reusable production test rig, keeps additional per-link optimization separate from adoption, and surfaces `prefetch={false}` links that may no longer be needed. - Applies the shared test-rig fixes discovered during dogfooding to the Cache Components optimizer. ## Why Legacy `prefetch={true}` could include uncached dynamic content. With Partial Prefetching, it includes the shared App Shell and cached URL-specific content instead. A passing build or clean insight sweep does not prove that the intended prefetched UI survived the migration, so the skill now records that contract before adoption and verifies it unchanged afterward. The guide presents the same test workflow as an optional programmatic check for manual adopters. ## Testing - Prettier, Alex, and skill frontmatter validation - Dogfooded on v0 in vercel/v0#28842 - Partial Prefetching adoption eval: 6/6 in #97759
Base automatically changed from
codex/partial-prefetch-preservation-guide
to
canary
August 27, 2026 19:46
gaojude
approved these changes
Aug 31, 2026
icyJoseph
pushed a commit
that referenced
this pull request
Sep 7, 2026
…ion (#97712) ## What - Documents how legacy `prefetch={true}` behavior changes with Partial Prefetching and how to choose the UI worth preserving. - Updates the adoption skill to capture a passing flag-off `instant()` baseline, use the same assertions to drive the migration, and retain them as regression coverage. - Adds a reusable production test rig, keeps additional per-link optimization separate from adoption, and surfaces `prefetch={false}` links that may no longer be needed. - Applies the shared test-rig fixes discovered during dogfooding to the Cache Components optimizer. ## Why Legacy `prefetch={true}` could include uncached dynamic content. With Partial Prefetching, it includes the shared App Shell and cached URL-specific content instead. A passing build or clean insight sweep does not prove that the intended prefetched UI survived the migration, so the skill now records that contract before adoption and verifies it unchanged afterward. The guide presents the same test workflow as an optional programmatic check for manual adopters. ## Testing - Prettier, Alex, and skill frontmatter validation - Dogfooded on v0 in vercel/v0#28842 - Partial Prefetching adoption eval: 6/6 in #97759 (cherry picked from commit 24ed66a)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Results
One
claude-opus-4-8run was retained per treatment and adoption eval, judged byclaude-haiku-4-5. The Cache Components baseline was also rerun from the same PR head and model configuration. These results are directional rather than statistically stable.The Partial Prefetching skill was the only treatment to establish a passing legacy
instant()baseline before migration, rerun the same production Playwright assertions after enabling Partial Prefetching, and preserve the selected eager-link contract. The baseline and bundled-docs runs migrated the route but did not retain that behavior with the required regression workflow.Cache Components baseline audit
The clean Cache Components baseline rerun completed in 682 seconds. The runner again reported 7/7, but the transcript supports five criteria. Two agentic checks were false positives:
<main>and a<Suspense fallback={null}>outside the URL-specific work, so it did not preserve useful visible shell content.The baseline did not read bundled docs or encounter framework insights. It inspected the installed Next.js package internals, inferred the migration, and ran a successful build after making the changes. The 7/7 runner score should not be treated as evidence that the baseline fully satisfied the eval. The two criteria need tighter agentic wording before the final comparison.
The current bundled-docs and skill scores remain 6/7 and 7/7. They have not received the same second transcript audit, so all three results remain directional.
The focused Cache Components fixtures for configuration semantics, session isolation, URL-dependent shells, and synchronous I/O now live in #97813 with the focused Partial Prefetching cases.
Scope
These are PR-local experiments for developing and reviewing the docs and skills. This PR does not register them in the public
evals.nextjs.orgsuite, which is configured separately in the OSS eval repository.Validation
pnpm exec tsc -p evals/tsconfig.json --noEmit