[feat] add flags to limit extraction

.prettierrc.json

@@ -0,0 +1,12 @@
+{
+  "$schema": "https://json.schemastore.org/prettierrc",
+  "semi": false,
+  "tabWidth": 2,
+  "singleQuote": true,
+  "printWidth": 100,
+  "trailingComma": "es5",
+  "importOrderSeparation": true,
+  "importOrderSortSpecifiers": true,
+  "bracketSameLine": false,
+  "htmlWhitespaceSensitivity": "ignore"
+}

README.md

@@ -3,15 +3,17 @@
 A streamlined CLI tool for creating new Directus projects and managing Directus templates - making it easy to apply and extract template configurations across instances.
 
 This tool is best suited for:
+
 - Proof of Concept (POC) projects
 - Demo environments
 - New project setups
 
 ⚠️ We strongly recommend against using this tool in existing production environments or as a critical part of your CI/CD pipeline without thorough testing. Always create backups before applying templates.
 
 **Important Notes:**
+
 - **Primary Purpose**: Built to deploy templates created by the Directus Core Team. While community templates are supported, the unlimited possible configurations make comprehensive support challenging.
-- **Database Compatibility**: PostgreSQL is recommended. Applying templates that are extracted and applied between different databases (Extract from SQLite ->  Apply to Postgres) can caused issues and is not recommended. MySQL users may encounter known issues.
+- **Database Compatibility**: PostgreSQL is recommended. Applying templates that are extracted and applied between different databases (Extract from SQLite -> Apply to Postgres) can caused issues and is not recommended. MySQL users may encounter known issues.
 - **Performance**: Remote operations (extract/apply) are rate-limited to 10 requests/second using bottleneck. Processing time varies based on your instance size (collections, items, assets).
 - **Version Compatibility**:
   - v0.5.0+: Compatible with Directus 11 and up
@@ -30,6 +32,7 @@ npx directus-template-cli@latest init
 ```
 
 You'll be guided through:
+
 - Selecting a directory for your new project
 - Choosing a Directus backend template
 - Selecting a frontend framework (if available for the template)
@@ -51,7 +54,6 @@ npx directus-template-cli@latest init my-project --frontend=nextjs --template=cm
 npx directus-template-cli@latest init --template=https://github.com/directus-labs/starters/tree/main/cms
 ```
 
-
 Available flags:
 
 - `--frontend`: Frontend framework to use (e.g., nextjs, nuxt, astro)
@@ -112,6 +114,7 @@ The `directus:template` property contains:
   - Each frontend has a `name` (display name) and `path` (directory containing the frontend code)
 
 When you use this template with the `init` command, it will:
+
 1. Copy the Directus template files from the specified template directory
 2. Copy the selected frontend code based on your choice or the `--frontend` flag
 3. Set up the project structure with both backend and frontend integrated
@@ -133,12 +136,10 @@ npx directus-template-cli@latest apply
 
 You can choose from our community maintained templates or you can also choose a template from a local directory or a public GitHub repository.
 
-
 ### Programmatic Mode
 
 By default, the CLI will run in interactive mode. For CI/CD pipelines or automated scripts, you can use the programmatic mode:
 
-
 Using a token:
 
 ```
@@ -196,7 +197,6 @@ This command will apply the template but exclude content and users. Available `-
 - `--no-settings`: Skip loading Settings
 - `--no-users`: Skip loading Users
 
-
 #### Template Component Dependencies
 
 When applying templates, certain components have dependencies on others. Here are the key relationships to be aware of:
@@ -231,7 +231,6 @@ You can also pass flags as environment variables. This can be useful for CI/CD p
 - `TEMPLATE_LOCATION`: Equivalent to `--templateLocation`
 - `TEMPLATE_TYPE`: Equivalent to `--templateType`
 
-
 ### Existing Data
 
 You can apply a template to an existing Directus instance. This is nice because you can have smaller templates that you can "compose" for various use cases. The CLI tries to be smart about existing items in the target Directus instance. But mileage may vary depending on the size and complexity of the template and the existing instance.
@@ -290,6 +289,10 @@ Available flags:
 - `--templateLocation`: Directory to extract the template to (required)
 - `--templateName`: Name of the template (required)
 - `--disableTelemetry`: Disable telemetry collection
+- `--skipCollectionFiles`: Do not extract the directus_files collection
+- `--skipDownloadFiles`: Do not download the actual file attachments
+- `--syncExtractContent`: Extract content collections in a serial manner to limit load on instance
+- `--limitContentCollections "collectionA,collectionB"`: Only extract named content collections from comma-separated list
 
 #### Using Environment Variables
 
@@ -306,9 +309,10 @@ Similar to the Apply command, you can use environment variables for the Extract
 The Directus Template CLI logs information to a file in the `.directus-template-cli/logs` directory.
 
 Logs are automatically generated for each run of the CLI. Here's how the logging system works:
-   - A new log file is created for each CLI run.
-   - Log files are stored in the `.directus-template-cli/logs` directory within your current working directory.
-   - Each log file is named `run-[timestamp].log`, where `[timestamp]` is the ISO timestamp of when the CLI was initiated.
+
+- A new log file is created for each CLI run.
+- Log files are stored in the `.directus-template-cli/logs` directory within your current working directory.
+- Each log file is named `run-[timestamp].log`, where `[timestamp]` is the ISO timestamp of when the CLI was initiated.
 
 The logger automatically sanitizes sensitive information such as passwords, tokens, and keys before writing to the log file. But it may not catch everything. Just be aware of this and make sure to remove the log files when they are no longer needed.
 

package.json

@@ -79,7 +79,8 @@
     "prepack": "pnpm run build && oclif manifest && oclif readme",
     "test": "mocha --forbid-only \"test/**/*.test.ts\"",
     "version": "oclif readme && git add README.md",
-    "run": "./bin/run.js"
+    "run": "./bin/run.js",
+    "run:dev": "./bin/dev.js"
   },
   "engines": {
     "node": ">=18.0.0"

src/commands/extract.ts

@@ -1,41 +1,58 @@
-import {text, select, intro, log} from '@clack/prompts'
-import {ux} from '@oclif/core'
+import { text, select, intro, log } from '@clack/prompts'
+import { ux } from '@oclif/core'
 import slugify from '@sindresorhus/slugify'
 import chalk from 'chalk'
 import fs from 'node:fs'
 import path from 'pathe'
 
 import * as customFlags from '../flags/common.js'
-import {DIRECTUS_PINK, DIRECTUS_PURPLE, SEPARATOR, BSL_LICENSE_TEXT, BSL_LICENSE_CTA, BSL_LICENSE_HEADLINE} from '../lib/constants.js'
-import {animatedBunny} from '../lib/utils/animated-bunny.js'
+import {
+  DIRECTUS_PINK,
+  DIRECTUS_PURPLE,
+  SEPARATOR,
+  BSL_LICENSE_TEXT,
+  BSL_LICENSE_CTA,
+  BSL_LICENSE_HEADLINE,
+} from '../lib/constants.js'
+import { animatedBunny } from '../lib/utils/animated-bunny.js'
 import { BaseCommand } from './base.js'
 import { track, shutdown } from '../services/posthog.js'
 
 import extract from '../lib/extract/index.js'
-import {getDirectusToken, getDirectusUrl, initializeDirectusApi, validateAuthFlags, getDirectusEmailAndPassword} from '../lib/utils/auth.js'
+import {
+  getDirectusToken,
+  getDirectusUrl,
+  initializeDirectusApi,
+  validateAuthFlags,
+  getDirectusEmailAndPassword,
+} from '../lib/utils/auth.js'
 import catchError from '../lib/utils/catch-error.js'
 import {
   generatePackageJsonContent,
   generateReadmeContent,
 } from '../lib/utils/template-defaults.js'
 
 export interface ExtractFlags {
-  directusToken: string;
-  directusUrl: string;
-  programmatic: boolean;
-  templateLocation: string;
-  templateName: string;
-  userEmail: string;
-  userPassword: string;
-  disableTelemetry?: boolean;
+  directusToken: string
+  directusUrl: string
+  programmatic: boolean
+  templateLocation: string
+  templateName: string
+  userEmail: string
+  userPassword: string
+  disableTelemetry?: boolean
+  skipCollectionFiles?: boolean
+  skipDownloadFiles?: boolean
+  syncExtractContent?: boolean
+  limitContentCollections?: string
 }
 
 export default class ExtractCommand extends BaseCommand {
   static description = 'Extract a template from a Directus instance.'
 
   static examples = [
     '$ directus-template-cli extract',
-    '$ directus-template-cli extract -p --templateName="My Template" --templateLocation="./my-template" --directusToken="admin-token-here" --directusUrl="http://localhost:8055"',
+    '$ directus-template-cli extract -p --templateName="My Template" --templateLocation="./my-template" --directusToken="admin-token-here" --directusUrl="http://localhost:8055" --skipDownloadFiles',
   ]
 
   static flags = {
@@ -47,17 +64,23 @@ export default class ExtractCommand extends BaseCommand {
     userEmail: customFlags.userEmail,
     userPassword: customFlags.userPassword,
     disableTelemetry: customFlags.disableTelemetry,
+    skipCollectionFiles: customFlags.skipCollectionFiles,
+    skipDownloadFiles: customFlags.skipDownloadFiles,
+    syncExtractContent: customFlags.syncExtractContent,
+    limitContentCollections: customFlags.limitContentCollections,
   }
 
   /**
    * Main run method for the ExtractCommand
    * @returns {Promise<void>} - Returns nothing
    */
   public async run(): Promise<void> {
-    const {flags} = await this.parse(ExtractCommand)
+    const { flags } = await this.parse(ExtractCommand)
     const typedFlags = flags as unknown as ExtractFlags
 
-    await (typedFlags.programmatic ? this.runProgrammatic(typedFlags) : this.runInteractive(typedFlags))
+    await (typedFlags.programmatic
+      ? this.runProgrammatic(typedFlags)
+      : this.runInteractive(typedFlags))
   }
 
   /**
@@ -67,10 +90,14 @@ export default class ExtractCommand extends BaseCommand {
    * @param {ExtractFlags} flags - The command flags
    * @returns {Promise<void>} - Returns nothing
    */
-  private async extractTemplate(templateName: string, directory: string, flags: ExtractFlags): Promise<void> {
+  private async extractTemplate(
+    templateName: string,
+    directory: string,
+    flags: ExtractFlags
+  ): Promise<void> {
     // Track start of extraction attempt
     if (!flags.disableTelemetry) {
-      await track({
+      track({
         command: 'extract',
         lifecycle: 'start',
         distinctId: this.userConfig.distinctId,
@@ -82,12 +109,12 @@ export default class ExtractCommand extends BaseCommand {
         },
         runId: this.runId,
         config: this.config,
-      });
+      })
     }
 
     try {
       if (!fs.existsSync(directory)) {
-        fs.mkdirSync(directory, {recursive: true})
+        fs.mkdirSync(directory, { recursive: true })
       }
 
       const packageJSONContent = generatePackageJsonContent(templateName)
@@ -100,23 +127,28 @@ export default class ExtractCommand extends BaseCommand {
       fs.writeFileSync(readmePath, readmeContent)
     } catch (error) {
       catchError(error, {
-        context: {function: 'extractTemplate'},
+        context: { function: 'extractTemplate' },
         fatal: true,
         logToFile: true,
       })
     }
 
     ux.stdout(SEPARATOR)
 
-    ux.action.start(`Extracting template - ${ux.colorize(DIRECTUS_PINK, templateName)} from ${ux.colorize(DIRECTUS_PINK, flags.directusUrl)} to ${ux.colorize(DIRECTUS_PINK, directory)}`)
+    ux.action.start(
+      `Extracting template - ${ux.colorize(DIRECTUS_PINK, templateName)} from ${ux.colorize(
+        DIRECTUS_PINK,
+        flags.directusUrl
+      )} to ${ux.colorize(DIRECTUS_PINK, directory)}`
+    )
 
-    await extract(directory)
+    await extract(directory, flags)
 
     ux.action.stop()
 
     // Track completion before final messages/exit
     if (!flags.disableTelemetry) {
-      await track({
+      track({
         command: 'extract',
         lifecycle: 'complete',
         distinctId: this.userConfig.distinctId,
@@ -128,8 +160,8 @@ export default class ExtractCommand extends BaseCommand {
         },
         runId: this.runId,
         config: this.config,
-      });
-      await shutdown();
+      })
+      await shutdown()
     }
 
     log.warn(BSL_LICENSE_HEADLINE)
@@ -147,7 +179,7 @@ export default class ExtractCommand extends BaseCommand {
    * @returns {Promise<void>} - Returns nothing
    */
   private async runInteractive(flags: ExtractFlags): Promise<void> {
-    await animatedBunny('Let\'s extract a template!')
+    await animatedBunny("Let's extract a template!")
 
     intro(`${chalk.bgHex(DIRECTUS_PURPLE).white.bold('Directus Template CLI')} - Extract Template`)
 
@@ -159,7 +191,8 @@ export default class ExtractCommand extends BaseCommand {
     const directory = await text({
       placeholder: `templates/${slugify(templateName as string)}`,
       defaultValue: `templates/${slugify(templateName as string)}`,
-      message: "What directory would you like to extract the template to? If it doesn't exist, it will be created.",
+      message:
+        "What directory would you like to extract the template to? If it doesn't exist, it will be created.",
     })
 
     ux.stdout(`You selected ${ux.colorize(DIRECTUS_PINK, directory as string)}`)
@@ -173,8 +206,8 @@ export default class ExtractCommand extends BaseCommand {
     // Prompt for login method
     const loginMethod = await select({
       options: [
-        {label: 'Directus Access Token', value: 'token'},
-        {label: 'Email and Password', value: 'email'},
+        { label: 'Directus Access Token', value: 'token' },
+        { label: 'Email and Password', value: 'email' },
       ],
       message: 'How do you want to log in?',
     })
@@ -183,7 +216,7 @@ export default class ExtractCommand extends BaseCommand {
       const directusToken = await getDirectusToken(directusUrl as string)
       flags.directusToken = directusToken as string
     } else {
-      const {userEmail, userPassword} = await getDirectusEmailAndPassword()
+      const { userEmail, userPassword } = await getDirectusEmailAndPassword()
       flags.userEmail = userEmail as string
       flags.userPassword = userPassword as string
     }
@@ -203,7 +236,7 @@ export default class ExtractCommand extends BaseCommand {
   private async runProgrammatic(flags: ExtractFlags): Promise<void> {
     this.validateProgrammaticFlags(flags)
 
-    const {templateLocation, templateName} = flags
+    const { templateLocation, templateName } = flags
 
     await initializeDirectusApi(flags)
 

src/flags/common.ts

@@ -1,4 +1,4 @@
-import {Flags} from '@oclif/core'
+import { Flags } from '@oclif/core'
 
 export const directusToken = Flags.string({
   description: 'Token to use for the Directus instance',
@@ -49,3 +49,26 @@ export const disableTelemetry = Flags.boolean({
   description: 'Disable telemetry',
   env: 'DISABLE_TELEMETRY',
 })
+
+export const skipCollectionFiles = Flags.boolean({
+  default: false,
+  description: 'Skip extracting collection "directus_files"',
+  env: 'SKIP_COLLECTION_FILES',
+})
+
+export const skipDownloadFiles = Flags.boolean({
+  default: false,
+  description: 'Skip downloading asset files',
+  env: 'SKIP_DOWNLOAD_FILES',
+})
+
+export const syncExtractContent = Flags.boolean({
+  default: false,
+  description: 'Fetch content collections synchronously to reduce load on the Directus instance',
+  env: 'SYNC_EXTRACT_CONTENT',
+})
+
+export const limitContentCollections = Flags.string({
+  description: 'Limit content extraction to specific collections (comma-separated)',
+  env: 'LIMIT_CONTENT_COLLECTIONS',
+})