Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add integrations API for db config/seed files #10321

Merged
merged 25 commits into from
Mar 7, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
e10d93c
Add integrations API for adding db config/seed files
delucis Mar 4, 2024
b80674d
Fix seeding when user seed file is present
delucis Mar 4, 2024
3ad617a
Add basic test and fixture for integrations API
delucis Mar 4, 2024
54a6054
Freeze that lockfile
delucis Mar 4, 2024
2148fc9
Test to see if this is a Windows fix
delucis Mar 4, 2024
d2b3d89
Don’t import.meta.glob integration seed files
delucis Mar 4, 2024
0ca265e
Make integration seed files export a default function
delucis Mar 5, 2024
985d3a0
style: rejiggle
delucis Mar 5, 2024
ef2156e
Fix temporary file conflicts
delucis Mar 5, 2024
ecd69ac
Remove changes to Astro’s core types, type utility method instead
delucis Mar 5, 2024
dd3e976
Merge branch 'main' into plt-1746/db-integrations
delucis Mar 5, 2024
29e09e0
Merge branch 'main' into plt-1746/db-integrations
delucis Mar 6, 2024
76b62b1
Use `astro:db` instead of `@astrojs/db`
delucis Mar 6, 2024
b2d2522
Merge branch 'main' into plt-1746/db-integrations
delucis Mar 7, 2024
76746e3
Merge branch 'main' into plt-1746/db-integrations
delucis Mar 7, 2024
0d0204a
Revert unnecessarily cautious temporary path name
delucis Mar 7, 2024
59edb2d
Add changeset
delucis Mar 7, 2024
20414ce
Fix entrypoints and `asDrizzleTable` usage in changeset
delucis Mar 7, 2024
2d3fdba
Merge branch 'main' into plt-1746/db-integrations
delucis Mar 7, 2024
915123a
Merge branch 'main' into plt-1746/db-integrations
delucis Mar 7, 2024
967513f
Getting Nate in on the co-author action
delucis Mar 7, 2024
3ba05b6
Fix user seed file in integrations fixture
delucis Mar 7, 2024
2a4e7d7
Merge branch 'main' into plt-1746/db-integrations
delucis Mar 7, 2024
1c06217
Update `seedLocal()` after merge
delucis Mar 7, 2024
6f6593b
Provide empty `seedFiles` array in `db execute`
delucis Mar 7, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 58 additions & 0 deletions .changeset/purple-poets-sin.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
---
"@astrojs/db": minor
---

Adds support for integrations providing `astro:db` configuration and seed files, using the new `astro:db:setup` hook.

To get TypeScript support for the `astro:db:setup` hook, wrap your integration object in the `defineDbIntegration()` utility:

```js
import { defineDbIntegration } from '@astrojs/db/utils';

export default function MyDbIntegration() {
return defineDbIntegration({
name: 'my-astro-db-powered-integration',
hooks: {
'astro:db:setup': ({ extendDb }) => {
extendDb({
configEntrypoint: '@astronaut/my-package/config',
seedEntrypoint: '@astronaut/my-package/seed',
});
},
},
});
}
```

Use the `extendDb` method to register additional `astro:db` config and seed files.

Integration config and seed files follow the same format as their user-defined equivalents. However, often while working on integrations, you may not be able to benefit from Astro’s generated table types exported from `astro:db`. For full type safety and autocompletion support, use the `asDrizzleTable()` utility to wrap your table definitions in the seed file.

```js
// config.ts
import { defineTable, column } from 'astro:db';

export const Pets = defineTable({
columns: {
name: column.text(),
age: column.number(),
},
});
```

```js
// seed.ts
import { asDrizzleTable } from '@astrojs/db/utils';
import { db } from 'astro:db';
import { Pets } from './config';

export default async function() {
// Convert the Pets table into a format ready for querying.
const typeSafePets = asDrizzleTable('Pets', Pets);

await db.insert(typeSafePets).values([
{ name: 'Palomita', age: 7 },
{ name: 'Pan', age: 3.5 },
]);
}
```
1 change: 1 addition & 0 deletions packages/db/src/core/cli/commands/execute/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@ export async function cmd({
tables: dbConfig.tables ?? {},
root: astroConfig.root,
shouldSeed: false,
seedFiles: [],
});
}
const { code } = await bundleFile({ virtualModContents, root: astroConfig.root, fileUrl });
Expand Down
7 changes: 2 additions & 5 deletions packages/db/src/core/cli/index.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import type { AstroConfig } from 'astro';
import type { Arguments } from 'yargs-parser';
import { loadDbConfigFile } from '../load-file.js';
import { dbConfigSchema } from '../types.js';
import { resolveDbConfig } from '../load-file.js';

export async function cli({
flags,
Expand All @@ -14,9 +13,7 @@ export async function cli({
// Most commands are `astro db foo`, but for now login/logout
// are also handled by this package, so first check if this is a db command.
const command = args[2] === 'db' ? args[3] : args[2];
const { mod } = await loadDbConfigFile(astroConfig.root);
// TODO: parseConfigOrExit()
const dbConfig = dbConfigSchema.parse(mod?.default ?? {});
const { dbConfig } = await resolveDbConfig(astroConfig);

switch (command) {
case 'shell': {
Expand Down
10 changes: 10 additions & 0 deletions packages/db/src/core/errors.ts
Original file line number Diff line number Diff line change
Expand Up @@ -71,3 +71,13 @@ export const FOREIGN_KEY_REFERENCES_EMPTY_ERROR = (tableName: string) => {
tableName
)} is misconfigured. \`references\` array cannot be empty.`;
};

export const INTEGRATION_TABLE_CONFLICT_ERROR = (
integrationName: string,
tableName: string,
isUserConflict: boolean
) => {
return red('▶ Conflicting table name in integration ' + bold(integrationName)) + isUserConflict
? `\n A user-defined table named ${bold(tableName)} already exists`
: `\n Another integration already added a table named ${bold(tableName)}`;
};
22 changes: 11 additions & 11 deletions packages/db/src/core/integration/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,22 +6,19 @@ import { mkdir, rm, writeFile } from 'fs/promises';
import { blue, yellow } from 'kleur/colors';
import parseArgs from 'yargs-parser';
import { CONFIG_FILE_NAMES, DB_PATH } from '../consts.js';
import { loadDbConfigFile } from '../load-file.js';
import { resolveDbConfig } from '../load-file.js';
import { type ManagedAppToken, getManagedAppTokenOrExit } from '../tokens.js';
import { type DBConfig, dbConfigSchema } from '../types.js';
import { type VitePlugin, getDbDirectoryUrl } from '../utils.js';
import { errorMap } from './error-map.js';
import { fileURLIntegration } from './file-url.js';
import { typegen } from './typegen.js';
import { type LateTables, vitePluginDb } from './vite-plugin-db.js';
import { type LateTables, vitePluginDb, type LateSeedFiles } from './vite-plugin-db.js';
import { vitePluginInjectEnvTs } from './vite-plugin-inject-env-ts.js';

function astroDBIntegration(): AstroIntegration {
let connectToStudio = false;
let configFileDependencies: string[] = [];
let root: URL;
let appToken: ManagedAppToken | undefined;
let dbConfig: DBConfig;

// Make table loading "late" to pass to plugins from `config:setup`,
// but load during `config:done` to wait for integrations to settle.
Expand All @@ -30,6 +27,11 @@ function astroDBIntegration(): AstroIntegration {
throw new Error('[astro:db] INTERNAL Tables not loaded yet');
},
};
let seedFiles: LateSeedFiles = {
get() {
throw new Error('[astro:db] INTERNAL Seed files not loaded yet');
},
};
let command: 'dev' | 'build' | 'preview';
return {
name: 'astro:db',
Expand Down Expand Up @@ -57,6 +59,7 @@ function astroDBIntegration(): AstroIntegration {
dbPlugin = vitePluginDb({
connectToStudio: false,
tables,
seedFiles,
root: config.root,
srcDir: config.srcDir,
});
Expand All @@ -74,13 +77,10 @@ function astroDBIntegration(): AstroIntegration {

// TODO: refine where we load tables
// @matthewp: may want to load tables by path at runtime
const { mod, dependencies } = await loadDbConfigFile(config.root);
const { dbConfig, dependencies, integrationSeedPaths } = await resolveDbConfig(config);
tables.get = () => dbConfig.tables;
seedFiles.get = () => integrationSeedPaths;
configFileDependencies = dependencies;
dbConfig = dbConfigSchema.parse(mod?.default ?? {}, {
errorMap,
});
// TODO: resolve integrations here?
tables.get = () => dbConfig.tables ?? {};

if (!connectToStudio) {
const dbUrl = new URL(DB_PATH, config.root);
Expand Down
20 changes: 18 additions & 2 deletions packages/db/src/core/integration/vite-plugin-db.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { type SQL, sql } from 'drizzle-orm';
import { SQLiteAsyncDialect } from 'drizzle-orm/sqlite-core';
Expand All @@ -23,11 +24,15 @@ const resolved = {
export type LateTables = {
get: () => DBTables;
};
export type LateSeedFiles = {
get: () => Array<string | URL>;
};

type VitePluginDBParams =
| {
connectToStudio: false;
tables: LateTables;
seedFiles: LateSeedFiles;
srcDir: URL;
root: URL;
}
Expand Down Expand Up @@ -81,6 +86,7 @@ export function vitePluginDb(params: VitePluginDBParams): VitePlugin {
return getLocalVirtualModContents({
root: params.root,
tables: params.tables.get(),
seedFiles: params.seedFiles.get(),
shouldSeed: id === resolved.seedVirtual,
});
},
Expand All @@ -94,17 +100,26 @@ export function getConfigVirtualModContents() {
export function getLocalVirtualModContents({
tables,
root,
seedFiles,
shouldSeed,
}: {
tables: DBTables;
seedFiles: Array<string | URL>;
root: URL;
shouldSeed: boolean;
}) {
const seedFilePaths = SEED_DEV_FILE_NAME.map(
const userSeedFilePaths = SEED_DEV_FILE_NAME.map(
// Format as /db/[name].ts
// for Vite import.meta.glob
(name) => new URL(name, getDbDirectoryUrl('file:///')).pathname
);
const resolveId = (id: string) => (id.startsWith('.') ? resolve(fileURLToPath(root), id) : id);
delucis marked this conversation as resolved.
Show resolved Hide resolved
const integrationSeedFilePaths = seedFiles.map((pathOrUrl) =>
typeof pathOrUrl === 'string' ? resolveId(pathOrUrl) : pathOrUrl.pathname
);
const integrationSeedImports = integrationSeedFilePaths.map(
(filePath) => `() => import(${JSON.stringify(filePath)})`
);
delucis marked this conversation as resolved.
Show resolved Hide resolved

const dbUrl = new URL(DB_PATH, root);
return `
Expand All @@ -117,7 +132,8 @@ export const db = createLocalDatabaseClient({ dbUrl });
${
shouldSeed
? `await seedLocal({
fileGlob: import.meta.glob(${JSON.stringify(seedFilePaths)}, { eager: true }),
userSeedGlob: import.meta.glob(${JSON.stringify(userSeedFilePaths)}, { eager: true }),
integrationSeedImports: [${integrationSeedImports.join(',')}],
});`
: ''
}
Expand Down
94 changes: 89 additions & 5 deletions packages/db/src/core/load-file.ts
Original file line number Diff line number Diff line change
@@ -1,12 +1,74 @@
import type { AstroConfig, AstroIntegration } from 'astro';
import { build as esbuild } from 'esbuild';
import { existsSync } from 'node:fs';
import { unlink, writeFile } from 'node:fs/promises';
import { fileURLToPath } from 'node:url';
import { build as esbuild } from 'esbuild';
import { createRequire } from 'node:module';
import { fileURLToPath, pathToFileURL } from 'node:url';
import { CONFIG_FILE_NAMES, VIRTUAL_MODULE_ID } from './consts.js';
import { INTEGRATION_TABLE_CONFLICT_ERROR } from './errors.js';
import { errorMap } from './integration/error-map.js';
import { getConfigVirtualModContents } from './integration/vite-plugin-db.js';
import { dbConfigSchema, type AstroDbIntegration } from './types.js';
import { getDbDirectoryUrl } from './utils.js';

export async function loadDbConfigFile(
const isDbIntegration = (integration: AstroIntegration): integration is AstroDbIntegration =>
'astro:db:setup' in integration.hooks;

/**
* Load a user’s `astro:db` configuration file and additional configuration files provided by integrations.
*/
export async function resolveDbConfig({ root, integrations }: AstroConfig) {
const { mod, dependencies } = await loadUserConfigFile(root);
const userDbConfig = dbConfigSchema.parse(mod?.default ?? {}, { errorMap });
/** Resolved `astro:db` config including tables provided by integrations. */
const dbConfig = { tables: userDbConfig.tables ?? {} };

// Collect additional config and seed files from integrations.
const integrationDbConfigPaths: Array<{ name: string; configEntrypoint: string | URL }> = [];
const integrationSeedPaths: Array<string | URL> = [];
for (const integration of integrations) {
if (!isDbIntegration(integration)) continue;
const { name, hooks } = integration;
if (hooks['astro:db:setup']) {
hooks['astro:db:setup']({
extendDb({ configEntrypoint, seedEntrypoint }) {
if (configEntrypoint) {
integrationDbConfigPaths.push({ name, configEntrypoint });
}
if (seedEntrypoint) {
integrationSeedPaths.push(seedEntrypoint);
}
},
});
}
}
for (const { name, configEntrypoint } of integrationDbConfigPaths) {
// TODO: config file dependencies are not tracked for integrations for now.
const loadedConfig = await loadIntegrationConfigFile(root, configEntrypoint);
const integrationDbConfig = dbConfigSchema.parse(loadedConfig.mod?.default ?? {}, {
errorMap,
});
for (const key in integrationDbConfig.tables) {
if (key in dbConfig.tables) {
const isUserConflict = key in (userDbConfig.tables ?? {});
throw new Error(INTEGRATION_TABLE_CONFLICT_ERROR(name, key, isUserConflict));
} else {
dbConfig.tables[key] = integrationDbConfig.tables[key];
}
}
}

return {
/** Resolved `astro:db` config, including tables added by integrations. */
dbConfig,
/** Dependencies imported into the user config file. */
dependencies,
/** Additional `astro:db` seed file paths provided by integrations. */
integrationSeedPaths,
};
}

async function loadUserConfigFile(
root: URL
): Promise<{ mod: { default?: unknown } | undefined; dependencies: string[] }> {
let configFileUrl: URL | undefined;
Expand All @@ -16,13 +78,35 @@ export async function loadDbConfigFile(
configFileUrl = fileUrl;
}
}
if (!configFileUrl) {
return await loadAndBundleDbConfigFile({ root, fileUrl: configFileUrl });
}

async function loadIntegrationConfigFile(root: URL, filePathOrUrl: string | URL) {
let fileUrl: URL;
if (typeof filePathOrUrl === 'string') {
const { resolve } = createRequire(root);
const resolvedFilePath = resolve(filePathOrUrl);
fileUrl = pathToFileURL(resolvedFilePath);
} else {
fileUrl = filePathOrUrl;
}
return await loadAndBundleDbConfigFile({ root, fileUrl });
}

async function loadAndBundleDbConfigFile({
root,
fileUrl,
}: {
root: URL;
fileUrl: URL | undefined;
}): Promise<{ mod: { default?: unknown } | undefined; dependencies: string[] }> {
if (!fileUrl) {
return { mod: undefined, dependencies: [] };
}
const { code, dependencies } = await bundleFile({
virtualModContents: getConfigVirtualModContents(),
root,
fileUrl: configFileUrl,
fileUrl,
});
return {
mod: await importBundledFile({ code, root }),
Expand Down
12 changes: 12 additions & 0 deletions packages/db/src/core/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { SQLiteAsyncDialect } from 'drizzle-orm/sqlite-core';
import { type ZodTypeDef, z } from 'zod';
import { SERIALIZED_SQL_KEY, type SerializedSQL } from '../runtime/types.js';
import { errorMap } from './integration/error-map.js';
import type { AstroIntegration } from 'astro';

export type MaybePromise<T> = T | Promise<T>;
export type MaybeArray<T> = T | T[];
Expand Down Expand Up @@ -271,3 +272,14 @@ export type ResolvedCollectionConfig<TColumns extends ColumnsConfig = ColumnsCon
// since Omit collapses our union type on primary key.
export type NumberColumnOpts = z.input<typeof numberColumnOptsSchema>;
export type TextColumnOpts = z.input<typeof textColumnOptsSchema>;

export type AstroDbIntegration = AstroIntegration & {
hooks: {
'astro:db:setup'?: (options: {
extendDb: (options: {
configEntrypoint?: URL | string;
seedEntrypoint?: URL | string;
}) => void;
}) => void | Promise<void>;
};
};
7 changes: 6 additions & 1 deletion packages/db/src/core/utils.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import type { AstroConfig } from 'astro';
import type { AstroConfig, AstroIntegration } from 'astro';
import { loadEnv } from 'vite';
import type { AstroDbIntegration } from './types.js';

export type VitePlugin = Required<AstroConfig['vite']>['plugins'][number];

Expand All @@ -21,3 +22,7 @@ export function getAstroStudioUrl(): string {
export function getDbDirectoryUrl(root: URL | string) {
return new URL('db/', root);
}

export function defineDbIntegration(integration: AstroDbIntegration): AstroIntegration {
return integration;
}
Loading
Loading