Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(page-data): add key to allPages #10625

Merged
merged 52 commits into from
May 8, 2024
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
9fcbe95
fix(page-data): add key to allPages
goulvenclech Mar 31, 2024
b6dd048
Merge branch 'main' into fix-all-pages-key
goulvenclech Apr 1, 2024
284ee9e
add fryuni's test
goulvenclech Apr 1, 2024
d42376c
replaced object.entries(allpages)
goulvenclech Apr 1, 2024
05f1ede
tmp: change pagesByComponents by pagesByKeys
goulvenclech Apr 1, 2024
db7bca5
fix pagesByKeys.get() in plugin-ssr & plugin-manifest
goulvenclech Apr 1, 2024
172623b
Merge branch 'main' into fix-all-pages-key
goulvenclech Apr 1, 2024
92f3099
remove logs
goulvenclech Apr 1, 2024
3021f42
remove useless generators
goulvenclech Apr 1, 2024
6853313
another useless generator
goulvenclech Apr 1, 2024
5db6dc6
Merge branch 'main' into fix-all-pages-key
goulvenclech Apr 1, 2024
1fd2690
Merge branch 'main' into fix-all-pages-key
goulvenclech Apr 9, 2024
8ef0d88
use null byte in key
goulvenclech Apr 9, 2024
98e9e90
tmp function in pipeline.ts
goulvenclech Apr 9, 2024
9306c71
refactor getVirtualModulePageName
goulvenclech Apr 9, 2024
03d4c59
refactor getPageKeyFromVirtualModulePageName
goulvenclech Apr 9, 2024
db7fdd2
Merge branch 'main' into fix-all-pages-key
goulvenclech Apr 9, 2024
21885cc
clean & comments
goulvenclech Apr 9, 2024
7e7469f
Merge branch 'main' into fix-all-pages-key
goulvenclech Apr 22, 2024
57137fd
better key and fix build
goulvenclech Apr 22, 2024
73ba151
utils: add makePageDataKey
goulvenclech Apr 22, 2024
96a6518
fix(pipeline): retrieveRoutesToGenerate for ssr
goulvenclech Apr 24, 2024
8793017
internals: getPageData function
goulvenclech Apr 24, 2024
8be75d4
Merge branch 'main' into fix-all-pages-key
goulvenclech Apr 24, 2024
ae2b017
tmp(ssr-split-manifest): fix test ?
goulvenclech Apr 25, 2024
b3ccdfb
fix?: ssr clean static output
goulvenclech Apr 25, 2024
2423ac2
internals: getPageDatasWithPublicKey
goulvenclech Apr 25, 2024
29da012
internals: getPageDatasByHoistedScriptId & getPagesDatasByComponent
goulvenclech Apr 25, 2024
cdbdcae
Merge branch 'main' into fix-all-pages-key
goulvenclech Apr 25, 2024
9b02b89
remove broken & useless virtualModuleNameFromResolvedId
goulvenclech Apr 26, 2024
cfec229
Merge branch 'main' into fix-all-pages-key
goulvenclech Apr 27, 2024
cd01367
chore: changeset
Princesseuh Apr 29, 2024
5c3a75f
fix: sanitize slashes in filepaths
Princesseuh Apr 29, 2024
896caf9
Revert "fix: sanitize slashes in filepaths"
goulvenclech Apr 29, 2024
7490882
fix?: remove route from virtual module name
goulvenclech Apr 29, 2024
c84d72c
fix: concat & array.from
goulvenclech Apr 29, 2024
2168f47
update changeset
goulvenclech Apr 29, 2024
984f6bf
clean unnecessary change
goulvenclech Apr 29, 2024
ced6e0f
remove unnecessary pageInfo
goulvenclech Apr 29, 2024
dfdfa7b
add return types to utils functions
goulvenclech Apr 29, 2024
cbf727e
revert a comment deletion
goulvenclech Apr 30, 2024
37ca1ae
fix cleanStaticOutput
goulvenclech Apr 30, 2024
846dd6f
changes from ematipico review
goulvenclech May 2, 2024
2c350d5
moving a todo outside jsdoc (cc @ematipico )
goulvenclech May 3, 2024
b424796
Update .changeset/great-turtles-greet.md
goulvenclech May 5, 2024
98d2364
Update .changeset/great-turtles-greet.md
goulvenclech May 5, 2024
1e17032
Merge remote-tracking branch 'origin/main' into fix-all-pages-key
ematipico May 8, 2024
8e2c178
Merge remote-tracking branch 'origin/main' into fix-all-pages-key
ematipico May 8, 2024
7477dc2
chore: fix merge conflicts
ematipico May 8, 2024
9d69309
fix: incorrect function
ematipico May 8, 2024
b8f3abc
remove logs
ematipico May 8, 2024
27395f4
revert: codepoint change
ematipico May 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 7 additions & 42 deletions packages/astro/src/core/build/internal.ts
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ export interface BuildInternals {
/**
* A map for page-specific information.
*/
pagesByComponent: Map<string, PageBuildData>;
pagesByKeys: Map<string, PageBuildData>;

/**
* A map for page-specific output.
Expand Down Expand Up @@ -125,7 +125,7 @@ export function createBuildInternals(): BuildInternals {
inlinedScripts: new Map(),
entrySpecifierToBundleMap: new Map<string, string>(),
pageToBundleMap: new Map<string, string>(),
pagesByComponent: new Map(),
pagesByKeys: new Map(),
pageOptionsByPage: new Map(),
pagesByViteID: new Map(),
pagesByClientOnly: new Map(),
Expand All @@ -151,7 +151,7 @@ export function trackPageData(
componentURL: URL
): void {
pageData.moduleSpecifier = componentModuleId;
internals.pagesByComponent.set(component, pageData);
internals.pagesByKeys.set(pageData.key, pageData);
internals.pagesByViteID.set(viteID(componentURL), pageData);
}

Expand Down Expand Up @@ -223,8 +223,9 @@ export function getPageDataByComponent(
internals: BuildInternals,
component: string
): PageBuildData | undefined {
if (internals.pagesByComponent.has(component)) {
return internals.pagesByComponent.get(component);
// TODO: Refactor that
if (internals.pagesByKeys.has(component)) {
return internals.pagesByKeys.get(component);
}
return undefined;
}
Expand All @@ -243,44 +244,8 @@ export function hasPageDataByViteID(internals: BuildInternals, viteid: ViteID):
return internals.pagesByViteID.has(viteid);
}

export function* eachPageData(internals: BuildInternals) {
yield* internals.pagesByComponent.values();
}

export function* eachPageFromAllPages(allPages: AllPagesData): Generator<[string, PageBuildData]> {
for (const [path, pageData] of Object.entries(allPages)) {
yield [path, pageData];
}
}

export function* eachPageDataFromEntryPoint(
internals: BuildInternals
): Generator<[PageBuildData, string]> {
for (const [entrypoint, filePath] of internals.entrySpecifierToBundleMap) {
// virtual pages can be emitted with different prefixes:
// - the classic way are pages emitted with prefix ASTRO_PAGE_RESOLVED_MODULE_ID -> plugin-pages
// - pages emitted using `build.split`, in this case pages are emitted with prefix RESOLVED_SPLIT_MODULE_ID
if (
entrypoint.includes(ASTRO_PAGE_RESOLVED_MODULE_ID) ||
entrypoint.includes(RESOLVED_SPLIT_MODULE_ID)
) {
const [, pageName] = entrypoint.split(':');
const pageData = internals.pagesByComponent.get(
`${pageName.replace(ASTRO_PAGE_EXTENSION_POST_PATTERN, '.')}`
);
if (!pageData) {
throw new Error(
"Build failed. Astro couldn't find the emitted page from " + pageName + ' pattern'
);
}

yield [pageData, filePath];
}
}
}

export function hasPrerenderedPages(internals: BuildInternals) {
for (const pageData of eachPageData(internals)) {
for (const pageData of internals.pagesByKeys.values()) {
if (pageData.route.prerender) {
return true;
}
Expand Down
10 changes: 6 additions & 4 deletions packages/astro/src/core/build/page-data.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ export async function collectPagesData(
// and is then cached across all future SSR builds. In the past, we've had trouble
// with parallelized builds without guaranteeing that this is called first.
for (const route of manifest.routes) {
// Generate a unique key to identify each page in the build process.
const key = `${route.route}_${route.component}`;
goulvenclech marked this conversation as resolved.
Show resolved Hide resolved
// static route:
if (route.pathname) {
const routeCollectionLogTimeout = setInterval(() => {
Expand All @@ -47,8 +49,8 @@ export async function collectPagesData(
clearInterval(routeCollectionLogTimeout);
}, 10000);
builtPaths.add(route.pathname);

allPages[route.component] = {
allPages[key] = {
key: key,
component: route.component,
route,
moduleSpecifier: '',
Expand All @@ -72,8 +74,8 @@ export async function collectPagesData(
continue;
}
// dynamic route:

allPages[route.component] = {
allPages[key] = {
key: key,
component: route.component,
route,
moduleSpecifier: '',
Expand Down
13 changes: 8 additions & 5 deletions packages/astro/src/core/build/pipeline.ts
Original file line number Diff line number Diff line change
Expand Up @@ -180,13 +180,13 @@ export class BuildPipeline extends Pipeline {
for (const [entrypoint, filePath] of this.internals.entrySpecifierToBundleMap) {
// virtual pages can be emitted with different prefixes:
// - the classic way are pages emitted with prefix ASTRO_PAGE_RESOLVED_MODULE_ID -> plugin-pages
// - pages emitted using `build.split`, in this case pages are emitted with prefix RESOLVED_SPLIT_MODULE_ID
// - pages emitted using `functionPerRoute`, in this case pages are emitted with prefix RESOLVED_SPLIT_MODULE_ID
if (
entrypoint.includes(ASTRO_PAGE_RESOLVED_MODULE_ID) ||
entrypoint.includes(RESOLVED_SPLIT_MODULE_ID)
) {
const [, pageName] = entrypoint.split(':');
const pageData = this.internals.pagesByComponent.get(
const pageData = this.internals.pagesByKeys.get(
`${pageName.replace(ASTRO_PAGE_EXTENSION_POST_PATTERN, '.')}`
);
if (!pageData) {
Expand All @@ -199,9 +199,9 @@ export class BuildPipeline extends Pipeline {
}
}

for (const [path, pageData] of this.internals.pagesByComponent.entries()) {
for (const pageData of this.internals.pagesByKeys.values()) {
if (routeIsRedirect(pageData.route)) {
pages.set(pageData, path);
pages.set(pageData, pageData.component);
} else if (
routeIsFallback(pageData.route) &&
(i18nHasFallback(this.config) ||
Expand All @@ -213,7 +213,10 @@ export class BuildPipeline extends Pipeline {
// The values of the map are the actual `.mjs` files that are generated during the build

// Here, we take the component path and transform it in the virtual module name
const moduleSpecifier = getVirtualModulePageNameFromPath(ASTRO_PAGE_MODULE_ID, path);
const moduleSpecifier = getVirtualModulePageNameFromPath(
ASTRO_PAGE_MODULE_ID,
pageData.component
);
// We retrieve the original JS module
const filePath = this.internals.entrySpecifierToBundleMap.get(moduleSpecifier);
if (filePath) {
Expand Down
5 changes: 2 additions & 3 deletions packages/astro/src/core/build/plugins/plugin-css.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ import {
moduleIsTopLevelPage,
} from '../graph.js';
import {
eachPageData,
getPageDataByViteID,
getPageDatasByClientOnlyID,
getPageDatasByHoistedScriptId,
Expand Down Expand Up @@ -214,7 +213,7 @@ function rollupPluginAstroBuildCSS(options: PluginOptions): VitePlugin[] {
(chunk) => chunk.type === 'asset' && chunk.name === 'style.css'
);
if (cssChunk === undefined) return;
for (const pageData of eachPageData(internals)) {
for (const pageData of internals.pagesByKeys.values()) {
const cssToInfoMap = (pagesToCss[pageData.moduleSpecifier] ??= {});
cssToInfoMap[cssChunk.fileName] = { depth: -1, order: -1 };
}
Expand Down Expand Up @@ -251,7 +250,7 @@ function rollupPluginAstroBuildCSS(options: PluginOptions): VitePlugin[] {
? { type: 'inline', content: stylesheet.source }
: { type: 'external', src: stylesheet.fileName };

const pages = Array.from(eachPageData(internals));
const pages = Array.from(internals.pagesByKeys.values());
let sheetAddedToPage = false;

pages.forEach((pageData) => {
Expand Down
5 changes: 4 additions & 1 deletion packages/astro/src/core/build/plugins/plugin-manifest.ts
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,10 @@ function buildManifest(
}

for (const route of opts.manifest.routes) {
const pageData = internals.pagesByComponent.get(route.component);
let pageData;
for (const page of internals.pagesByKeys.values()) {
if (page.route.route == route.route) pageData = page;
}
goulvenclech marked this conversation as resolved.
Show resolved Hide resolved
if (route.prerender || !pageData) continue;
const scripts: SerializedRouteInfo['scripts'] = [];
if (pageData.hoistedScript) {
Expand Down
19 changes: 15 additions & 4 deletions packages/astro/src/core/build/plugins/plugin-pages.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import type { Plugin as VitePlugin } from 'vite';
import { routeIsRedirect } from '../../redirects/index.js';
import { addRollupInput } from '../add-rollup-input.js';
import { type BuildInternals, eachPageFromAllPages } from '../internal.js';
import { type BuildInternals } from '../internal.js';
import type { AstroBuildPlugin } from '../plugin.js';
import type { StaticBuildOptions } from '../types.js';
import { RENDERERS_MODULE_ID } from './plugin-renderers.js';
Expand All @@ -22,11 +22,11 @@ function vitePluginPages(opts: StaticBuildOptions, internals: BuildInternals): V
if (opts.settings.config.output === 'static') {
const inputs = new Set<string>();

for (const [path, pageData] of eachPageFromAllPages(opts.allPages)) {
for (const pageData of Object.values(opts.allPages)) {
if (routeIsRedirect(pageData.route)) {
continue;
}
inputs.add(getVirtualModulePageNameFromPath(ASTRO_PAGE_MODULE_ID, path));
inputs.add(getVirtualModulePageNameFromPath(ASTRO_PAGE_MODULE_ID, pageData.component));
}

return addRollupInput(options, Array.from(inputs));
Expand All @@ -42,7 +42,9 @@ function vitePluginPages(opts: StaticBuildOptions, internals: BuildInternals): V
const imports: string[] = [];
const exports: string[] = [];
const pageName = getPathFromVirtualModulePageName(ASTRO_PAGE_RESOLVED_MODULE_ID, id);
const pageData = internals.pagesByComponent.get(pageName);
// TODO: Change that
// const pageData = internals.pagesByKeys.get(pageName);
const pageData = tmp(internals.pagesByKeys, pageName);
if (pageData) {
const resolvedPage = await this.resolve(pageData.moduleSpecifier);
if (resolvedPage) {
Expand Down Expand Up @@ -72,3 +74,12 @@ export function pluginPages(opts: StaticBuildOptions, internals: BuildInternals)
},
};
}

/**
* TMP: This is a temporary function to get the page data from the pagesByKeys map.
*/
function tmp(pagesByKeys: Map<string, any>, pageName: string) {
for (const pages of pagesByKeys.values()) {
if (pages.component == pageName) return pages;
}
}
18 changes: 10 additions & 8 deletions packages/astro/src/core/build/plugins/plugin-ssr.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ import { isServerLikeOutput } from '../../../prerender/utils.js';
import { routeIsRedirect } from '../../redirects/index.js';
import { addRollupInput } from '../add-rollup-input.js';
import type { BuildInternals } from '../internal.js';
import { eachPageFromAllPages } from '../internal.js';
import type { AstroBuildPlugin } from '../plugin.js';
import type { StaticBuildOptions } from '../types.js';
import { SSR_MANIFEST_VIRTUAL_MODULE_ID } from './plugin-manifest.js';
Expand Down Expand Up @@ -44,18 +43,21 @@ function vitePluginSSR(
let i = 0;
const pageMap: string[] = [];

for (const [path, pageData] of eachPageFromAllPages(allPages)) {
for (const pageData of Object.values(allPages)) {
if (routeIsRedirect(pageData.route)) {
continue;
}
const virtualModuleName = getVirtualModulePageNameFromPath(ASTRO_PAGE_MODULE_ID, path);
const virtualModuleName = getVirtualModulePageNameFromPath(
ASTRO_PAGE_MODULE_ID,
pageData.component
);
let module = await this.resolve(virtualModuleName);
if (module) {
const variable = `_page${i}`;
// we need to use the non-resolved ID in order to resolve correctly the virtual module
imports.push(`const ${variable} = () => import("${virtualModuleName}");`);

const pageData2 = internals.pagesByComponent.get(path);
const pageData2 = internals.pagesByKeys.get(pageData.key);
if (pageData2) {
pageMap.push(`[${JSON.stringify(pageData2.component)}, ${variable}]`);
}
Expand Down Expand Up @@ -147,11 +149,11 @@ function vitePluginSSRSplit(
if (functionPerRouteEnabled) {
const inputs = new Set<string>();

for (const [path, pageData] of eachPageFromAllPages(options.allPages)) {
for (const pageData of Object.values(options.allPages)) {
if (routeIsRedirect(pageData.route)) {
continue;
}
inputs.add(getVirtualModulePageNameFromPath(SPLIT_MODULE_ID, path));
inputs.add(getVirtualModulePageNameFromPath(SPLIT_MODULE_ID, pageData.component));
}

return addRollupInput(opts, Array.from(inputs));
Expand Down Expand Up @@ -293,8 +295,8 @@ function storeEntryPoint(
fileName: string
) {
const componentPath = getPathFromVirtualModulePageName(RESOLVED_SPLIT_MODULE_ID, moduleKey);
for (const [page, pageData] of eachPageFromAllPages(options.allPages)) {
if (componentPath == page) {
for (const pageData of Object.values(options.allPages)) {
if (componentPath == pageData.component) {
const publicPath = fileURLToPath(options.settings.config.build.server);
internals.entryPoints.set(pageData.route, pathToFileURL(join(publicPath, fileName)));
}
Expand Down
23 changes: 9 additions & 14 deletions packages/astro/src/core/build/static-build.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,7 @@ import * as vite from 'vite';
import type { RouteData } from '../../@types/astro.js';
import { PROPAGATED_ASSET_FLAG } from '../../content/consts.js';
import { hasAnyContentFlag } from '../../content/utils.js';
import {
type BuildInternals,
createBuildInternals,
eachPageData,
} from '../../core/build/internal.js';
import { type BuildInternals, createBuildInternals } from '../../core/build/internal.js';
import { emptyDir, removeEmptyDirs } from '../../core/fs/index.js';
import { appendForwardSlash, prependForwardSlash, removeFileExtension } from '../../core/path.js';
import { isModeServerWithNoAdapter } from '../../core/util.js';
Expand All @@ -37,7 +33,6 @@ import { encodeName, getTimeStat, viteBuildReturnToRollupOutputs } from './util.

export async function viteBuild(opts: StaticBuildOptions) {
const { allPages, settings } = opts;

// Make sure we have an adapter before building
if (isModeServerWithNoAdapter(opts.settings)) {
throw new AstroError(AstroErrorData.NoAdapterInstalled);
Expand All @@ -46,17 +41,18 @@ export async function viteBuild(opts: StaticBuildOptions) {
settings.timer.start('SSR build');

// The pages to be built for rendering purposes.
// (comment above may be outdated ?)
const pageInput = new Set<string>();

// Build internals needed by the CSS plugin
const internals = createBuildInternals();

for (const [component, pageData] of Object.entries(allPages)) {
const astroModuleURL = new URL('./' + component, settings.config.root);
const astroModuleId = prependForwardSlash(component);
for (const pageData of Object.values(allPages)) {
const astroModuleURL = new URL('./' + pageData.component, settings.config.root);
const astroModuleId = prependForwardSlash(pageData.component);

// Track the page data in internals
trackPageData(internals, component, pageData, astroModuleId, astroModuleURL);
trackPageData(internals, pageData.component, pageData, astroModuleId, astroModuleURL);

if (!routeIsRedirect(pageData.route)) {
pageInput.add(astroModuleId);
Expand All @@ -73,7 +69,6 @@ export async function viteBuild(opts: StaticBuildOptions) {
// Register plugins
const container = createPluginContainer(opts, internals);
registerAllPlugins(container);

// Build your project (SSR application code, assets, client JS, etc.)
const ssrTime = performance.now();
opts.logger.info('build', `Building ${settings.config.output} entrypoints...`);
Expand Down Expand Up @@ -271,7 +266,7 @@ async function ssrBuild(

const updatedViteBuildConfig = await runHookBuildSetup({
config: settings.config,
pages: internals.pagesByComponent,
pages: internals.pagesByKeys,
vite: viteBuildConfig,
target: 'server',
logger: opts.logger,
Expand Down Expand Up @@ -332,7 +327,7 @@ async function clientBuild(

await runHookBuildSetup({
config: settings.config,
pages: internals.pagesByComponent,
pages: internals.pagesByKeys,
vite: viteBuildConfig,
target: 'client',
logger: opts.logger,
Expand Down Expand Up @@ -373,7 +368,7 @@ async function cleanStaticOutput(
ssrOutputChunkNames: string[]
) {
const allStaticFiles = new Set();
for (const pageData of eachPageData(internals)) {
for (const pageData of internals.pagesByKeys.values()) {
if (pageData.route.prerender && !pageData.hasSharedModules) {
const { moduleSpecifier } = pageData;
const pageBundleId = internals.pageToBundleMap.get(moduleSpecifier);
Expand Down
1 change: 1 addition & 0 deletions packages/astro/src/core/build/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ export type StylesheetAsset =
| { type: 'external'; src: string };

export interface PageBuildData {
key: string;
component: ComponentPath;
route: RouteData;
moduleSpecifier: string;
Expand Down
Loading
Loading