Feature request: Add deno generate subcommand

[EthanThatOneKid]

Code generation has long been a valuable tool for developers, enabling us to automate repetitive tasks and improve the quality of our code. The introduction of the deno generate subcommand to the Deno toolchain has the potential to make code generation even more accessible and powerful. This feature could help developers to save time and effort, and to produce more consistent and reliable code.

Design overview

Add a deno generate subcommand to automate code generation by running procedures defined in comment annotations. The deno generate subcommand should scan entrypoint files for directives, which are lines starting with the comment:

//deno:generate <command> [arguments...]

Where command is the generator corresponding to an executable file that can be run locally. Arguments are passed to the generator as command line arguments.

When the deno generate command is executed, module graphs are initialized for each valid entry point. The deno generate subcommand will then scan each module graph for directives. If a directive is found, the generator is run with the arguments provided in the directive. The generator is run from the directory containing the directive.

Desired features

• Allow users to run deno generate and apply the command on each valid Deno entry point file in the current directory if no entry points are listed. Allow users to list space-separated files as entry points.
• Allow users to set a command alias for the rest of the file using the //deno:generate -command <command> [arguments...] directive.
• Allow users to generate using the LSP (meaning they can execute //deno:generate commands in VSCode).

• Each //deno:generate generator runs independently, and the failure of one command doesn't stop or impact the execution of other commands. A failed command generates an error message or exit code for itself without affecting the rest of the commands.
• Allow users to pass the following flags to deno generate:
  • --files to specify which files to ignore and include.
  • --run to specify a command to run.
  • --skip to specify a command to skip. In the event that a command matches both the --run and --skip regular expressions, the command is skipped.
  • --dry-run to print the commands that would be run without actually running them.
  • --verbose to print the module specifier and directive text of each directive when running the corresponding generator.
  • --trace to print the commands as they are run.

Help text

I have drafted the following help text for the deno generate subcommand:

Automate code generation by running procedures defined in comment annotations.

Usage:
  deno generate [<file>...] [options]

Options:
  -n, --dry-run            Print the commands that would be run without 
                           actually running them.
  -r, --run <regexp>       Select directives to run by matching against the
                           directive text as-is.
  -s, --skip <regexp>      Select directives to skip by matching against the
                           directive text as-is.
  -v, --verbose            Print the module specifier and directive text of
                           each directive when running the corresponding
                           generator.
  -x, --trace              Print the commands as they are run.

Examples:
  deno generate
  deno generate myfile1.ts
  deno generate myfile1.ts myfile2.ts
  deno generate myfile1.ts --dry-run

Deno generate scans a module graph for directives, which are lines starting
with the comment:

  //deno:generate <command> [arguments...]

where command is the generator corresponding to an executable file that can be
run locally. To run, arguments are passed to the generator. The generator is run
from the directory containing the directive.

Note: No space in between "//" and "deno:generate".

The deno generate command does not parse the file, so lines that look like
directives in comments or strings will be treated as directives.

The arguments to the directive are space-separated tokens or double-quoted
strings passed to the generator as arguments. Quoted strings are evaluated
before execution.

To convey to humans and tools that code is generated, generated source files
should have a comment of the form:

  ^// Code generated .* DO NOT EDIT\.$

The line must appear before the first non-comment, non-blank line of the file.

Deno generate sets several variables when running generators:

  $DENO_OS           The operating system of the host running deno generate.
  $DENO_MODULE       The module specifier of the module containing the directive.
  $DENO_LINE         The line number of the directive within the file.
  $DENO_CHARACTER    The character number of the directive within the file.
  $DENO_DIR          The directory containing the file containing the directive.
  $DOLLAR            A dollar sign ($). This is useful for escaping the $ in shell commands.
                     Literature: https://go-review.googlesource.com/c/go/+/8091

A directive may define a command alias for the file:

  //deno:generate -command <command> [arguments...]

where, for the remainder of this source file, the command <command> is replaced
by the arguments. This can be used to create aliases for long commands or to
define arguments that are common to multiple directives. For example:

  //deno:generate -command cat deno run --allow-read https://deno.land/std/examples/cat.ts
  (...)
  //deno:generate deno run ./generate_code.ts

The -command directive may appear anywhere in the file, but it is usually placed
at the top of the file, before any directives that use it.

The --run flag specifies a regular expression to select directives to run by
matching against the directive text as-is. The regular expression does not need
to match the entire text of the directive, but it must match at least one
character. The default is to run all directives.

The --skip flag specifies a regular expression to select directives to skip by
matching against the directive text as-is. The regular expression does not need
to match the entire text of the directive, but it must match at least one
character. The default is to not skip any directives.

The --dry-run flag (-n) prints the commands that would be run without actually
running them.

The --verbose flag (-v) prints the module specifier and directive
text of each directive when running the corresponding generator.

The --trace flag (-x) prints the commands as they are run.

You can also combine these flags to modify deno generate's behavior in different
ways. For example, deno generate -n -v mod.ts will run generate in dry run mode
and print more detailed information about the commands that it would run, while
deno generate -v -x will run generate in verbose mode and print the commands
that it is running as it runs them.

Conventions

Pre-commit

To enhance your development workflow, we recommend implementing a pre-commit hook in your project's Git repository. Follow these steps to set it up:

1. Create a file named "pre-commit" (without any file extension) within your project's ".git/hooks" directory.
2. Ensure that the file is executable by running the following command in your terminal:

chmod +x .git/hooks/pre-commit

3. Open the "pre-commit" file in a text editor and add the following code:

#!/bin/bash

# Run 'deno generate' before committing
deno generate

# Check if 'deno generate' generated any changes
git diff --exit-code

Linguist generated

When dealing with code generation, there are situations where generated files should not be visible to developers during a pull request. To address this, a setting can be used to differentiate their changes, ensuring a cleaner and more focused code review. On GitHub, you can achieve this by marking specific files with the "linguist-generated" attribute in a ".gitattributes" file¹. This attribute allows you to hide these files by default in diffs and exclude them from contributing to the repository language statistics.

To implement this, follow these steps:

1. Create a ".gitattributes" file in your project's root directory if it doesn't already exist.
2. Open the ".gitattributes" file in a text editor and include the relevant file patterns along with the "linguist-generated" attribute. For example:

# Marking generated files
*.generated.extension linguist-generated

3. Save the file and commit it to your repository.

Now, when viewing pull requests or generating diffs on GitHub, the marked files will be hidden by default, providing a more streamlined code review process and excluding them from language statistics calculations.

Use cases

The deno generate subcommand is a powerful feature, encouraging the use of code generation with a convenient developer experience. Code generation is an important programming technique because generated files can automate repetitive or complex tasks, improve code consistency and maintainability, and save developers time and effort.

As for use cases, your imagination is the limit. Here are a few:

1. Generating code from templates: Developers can define templates for commonly used code patterns and use the deno generate subcommand to automatically generate code that follows those patterns.
2. Generating code from schemas: If a project uses a schema to define data models or API specifications, developers can create generators that generate code based on that schema.
3. Generating tests: Developers can define test templates that cover common testing scenarios and use the deno generate subcommand to automatically generate tests for their code.
4. Generating code from annotations: Developers can add annotations to their code that define which generators to run and how to run them.

Usage

The deno generate subcommand is capable of facilitating the generation of code in the Deno ecosystem. Developers use code generation to generate code for all layers of your application.

Run any Deno script

Deno scripts should be able to invoke another Deno in its //deno:generate statement:

generator.ts

//deno:generate deno run -A generate.ts
for (let i = 0; i < 10; i++) {
  console.log(`export const example${i} = ${i};`);
}

generate.ts

// Create a child process using Deno.Command, running the "generate.ts" script.
const generatorChild = new Deno.Command(Deno.execPath(), {
  args: ["run", "generator.ts"],
  stdin: "piped",
  stdout: "piped",
}).spawn();

// Create another child process, running deno fmt.
const fmtChild = new Deno.Command(Deno.execPath(), {
  args: ["fmt", "-"],
  stdin: "piped",
  stdout: "piped",
}).spawn();

// Pipe the current process stdin to the child process stdin.
generatorChild.stdout.pipeTo(fmtChild.stdin);

// Close the child process stdin.
generatorChild.stdin.close();

// Pipe the child process stdout to a writable file named "generated.ts".
fmtChild.stdout.pipeTo(
  Deno.openSync("generated.ts", { write: true, create: true }).writable,
);

Generate OpenAPI types

OpenAPI is a JSON-based specification that represents comprehensive API details, providing a formal and professional representation of the API specifications. One of the most common use cases of OpenAPI is to generate API clients, simplifying the development process by automatically generating code based on the defined API specifications. OpenAPI schemas offer versatile applications and integrations, enabling a wide range of possibilities for API design and development.

//deno:generate deno run -A npm:openapi-typescript@6.2.4 ./examples/github_api.json -o ./examples/github_api.ts

Generate static website with Lume

Lume is a website framework for the Deno ecosystem. Entire static websites are generated by Lume with a single command. See more.

//deno:generate deno run -A https://deno.land/x/lume@v1.17.3/cli.ts --src ./examples/lume --dest ./examples/lume/_site

Generate deno_bindgen bindings

This tool aims to simplify glue code generation for Deno FFI libraries written in Rust. See more.

//deno:generate deno run -A https://deno.land/x/deno_bindgen@0.8.0/cli.ts

Generate deno-embedder file

deno-embedder is a tool that simplifies the development and distribution of Deno applications, particularly when access to static files (.txt, .png, etc.) is required at runtime. It allows you to create an embedder.ts file that encompasses both configuration and the main function call, providing benefits such as IDE-based type-checking. See more.

mod.ts

//deno:generate deno run -A embedder.ts
import staticFiles from "./embed/static/dir.ts";

const indexHTML = await staticFiles.load("index.html");
console.log("index.html:", await indexHTML.text());

Bundlee is a deno-embedder alternative.

import { Bundlee } from "https://deno.land/x/bundlee/mod.ts";

//deno:generate deno run -A https://deno.land/x/bundlee@0.9.4/bundlee.ts --bundle static/ bundle.json
const staticFiles = await Bundlee.load("bundle.json");

Generate deno doc JSON

mod.ts

//deno:generate deno run -A generate_docs.ts
import doc from "./doc.json" assert { type: "json" };

generate_docs.ts

// Create a child process running `deno doc --json`.
const child = new Deno.Command(Deno.execPath(), {
  args: ["doc", "--json"],
  stdin: "piped",
  stdout: "piped",
}).spawn();

// Pipe the child process stdout to a writable file named "doc.json".
child.stdout.pipeTo(
  Deno.openSync("doc.json", { write: true, create: true }).writable,
);

Generate art

mod.ts

//deno:generate deno run -A generate_art.ts
const art = Deno.readTextFileSync("art.txt");

generate_art.ts

// Create a child process running terminal_images.
const child = new Deno.Command(Deno.execPath(), {
  args: [
    "run",
    "-A",
    "https://deno.land/x/terminal_images@3.1.0/cli.ts",
    "https://deno.land/images/artwork/hashrock_simple.png",
  ],
  stdin: "piped",
  stdout: "piped",
}).spawn();

// Pipe the child process stdout to a writable file named "static/doc.json".
child.stdout.pipeTo(
  Deno.openSync("art.txt", { write: true, create: true }).writable,
);

Implications on permissions

The deno generate feature allows us to run our Deno programs in a slightly different way. Normally, when we use "deno run" to execute our code, we have to explicitly specify the permissions we need, like deno run --allow-etc. or deno run -A. However, with deno generate, we can run our program without explicitly mentioning the permissions.

It's important to note that this doesn't change how our program works or behaves. It's simply a different way of running it. However, because Deno is very careful about permissions, it's worth mentioning this aspect.

Ideally, only developers should run deno generate. This ensures that we have a clear understanding and control over the permissions granted during the development process.

Closing remarks

It is recommended that educational materials be added to Deno to clearly communicate to developers what types of generators they are encouraged and warned to run via the deno generate subcommand. This is important because Deno is a powerful tool that can be used to create a variety of applications, and it is important for developers to be aware of the potential risks and benefits of using different generators. By adding educational materials, the Deno team can help to ensure that developers are using the tool safely and effectively.

Overall, the deno generate subcommand has the potential to improve the development experience for Deno developers and attract new users to the ecosystem. Looking forward to the future of Deno!

Footnotes

1. https://docs.github.com/en/repositories/working-with-files/managing-files/customizing-how-changed-files-appear-on-github ↩

[dsherret]

This feels a lot like a task runner, but we already have deno task. Do you think perhaps this could be rolled into a script in userland, then people could do deno task generate?

[EthanThatOneKid]

This feels a lot like a task runner, but we already have deno task. Do you think perhaps this could be rolled into a script in userland, then people could do deno task generate?

I think is very doable to at least move forward with deno task generate. It sounds like a fun project, and I am more than happy to work on it. This way, other developers will be able to easily adopt the convention and benefit from it.

[EthanThatOneKid]

The userland script is now available under https://deno.land/x/generate! Check it out:

deno task generate

deno.jsonc

{
  "tasks": {
    "generate": "deno run -Ar https://deno.land/x/generate/cli/main.ts --verbose generator.ts"
  }
}

generator.ts

The provided code generates basic TypeScript code using a for loop to export
constants.

//deno:generate deno run -A generate.ts
for (let i = 0; i < 10; i++) {
  console.log(`export const example${i} = ${i};`);
}

generate.ts

// Create a child process using Deno.Command, running the "generate.ts" script.
const generatorChild = new Deno.Command(Deno.execPath(), {
  args: ["run", "generator.ts"],
  stdin: "piped",
  stdout: "piped",
}).spawn();

// Create another child process, running deno fmt.
const fmtChild = new Deno.Command(Deno.execPath(), {
  args: ["fmt", "-"],
  stdin: "piped",
  stdout: "piped",
}).spawn();

// Pipe the current process stdin to the child process stdin.
generatorChild.stdout.pipeTo(fmtChild.stdin);

// Close the child process stdin.
generatorChild.stdin.close();

// Pipe the child process stdout to a writable file named "generated.ts".
fmtChild.stdout.pipeTo(
  Deno.openSync("generated.ts", { write: true, create: true }).writable,
);

[elycheikhsmail]

@EthanThatOneKid I'm agree there is need for generating code, myself I created small tool for this perpose: https://github.com/elycheikhsmail/templatify-code.
But should this tool implemented in deno cli ? or as third party library ? or in deno std ?

[EthanThatOneKid]

I'm agree there is need for generating code, myself I created small tool for this perpose: https://github.com/elycheikhsmail/templatify-code.
But should this tool implemented in deno cli ? or as third party library ? or in deno std ?

Thank you for sharing your project templatify-code, @elycheikhsmail! templatify-code is a tool which seems to be compatible with //deno:generate.

Given that Deno draws inspiration from Go, I believe a deno generate subcommand should be added to the official Deno CLI. In the Go community, go generate brings about convenience and wider adoption of code generation.

[jeff-hykin]

As someone who embeds stuff often, I've definitely considered features like this. In this proposals current form, I think/agree it would be best as a 3rd party module.

However, I do think there is something more minimal/generic that is worth proposing as a change to deno.

For code generation and file embedding to work seamlessly, I believe we need an at-compile-time hook. JSX is already effectively a compile time hook. But in terms of custom hooks, AFAIK it is currently not possible to have deno run some code (or command) whenever deno bundle or deno compile is run, aside from making some hacky shell wrapper around the deno command that intercepts the compile command.

A Deno compile time hook would combine nicely with a 3rd party deno-generate command.

My recommended syntax would be similar to the JSX hook:

/** @deno:onCompile
 *
 *  import { run } from "deno.land/x/deno-generate"
 *  run`deno run -A generate.ts`
 */

[guy-borderless]

Wanted to pitch in a suggestion.
Use case: I use utility functions to parse different measures written in different systems, depending mostly on locale, over 300 such utility functions and counting. I don't want to pass the locale for each call (locale is the same for all calls in a specific context). I also don't want to use a generator function for all the code and lose tree shaking.

It would be great if a deno native code generator would tie in with ESM module resolution and make it easy for me to import utils.ts?locale=en-US or ./en-US/utils.ts?generate=true . It would JIT generate only imported code variants and cache them, making code-gen maintenance easier than current solutions.