[RFC] Code Prompts, a Prompt Format for AI-Powered Code Auto-Completion

[DanielRosenwasser]

A Prompt Format For AI-Powered Code Auto-Completion

A few years ago, I invented GitHub Copilot Chat and ChatGPT. While this was really hard to think of, I think it turned out really really good. On the other hand, using a chat bot to write code is kind of annoying sometimes - so maybe sometimes it's bad. That's why a lot of the time, I prefer Copilot's inline completions more. I know, nobody else does, but I do. I think.

The problem is that language models don't always know what code is valid. As a result, developers often need to review the code and find all the issues. While pointing out the mistakes of an AI helps bolster our self-esteem, we developers could be doing so much more. We need to improve the success rate of these language models to make them effective.

One key insight is that developers want to stop being software engineers, and instead become prompt engineers. Developers are infamously sick of learning new libraries, techniques and frameworks. They want only to be concerned with planning, processes, and the business bottom line. They want deeply to find the best way to tell an AI how to do everything so they can achieve this.

That's why today I'm proposing a new format to aid auto-completion tools: code prompts.

"Code Prompts?"

Yes, is there an echo in here?

Code prompts are a succinct declarative format to guide AI language models to write valid code. Rather than guessing based on surrounding code context, code prompts add rich information to prime a language model for inline completions.

For example, consider a variable named person. Imagine it has a name and an age property, along with a method called syncToDatabase(). As a developer, I want the AI to increment the age property and sync that to a database. Note that syncToDatabase is an asynchronous operation, and we want any subsequent code to be aware of this.

Traditionally, describing this to a language model was verbose and cumbersome. In fact, I basically just did to you. Here's what it would look like in code.

// consider a variable named `person`. Imagine it has a `name` and an `age` property
// along with a method called `syncToDatabase()`. As a developer, I want the AI to
// increment the age property and sync that to a database. Note that `syncToDatabase` is an
// asynchronous operation, and we want any subsequent code to be aware of this.
// Also, name is a string and age is a number.

This is super duper annoying to read, and at least twice as annoying to write (especially since I just wrote it twice).

Code prompts help dramatically here. Before we start off, I want you to be aware of how easy it is to add code prompts to an existing prompt.

First, let's start small.

// Imagine `person` has a `name` and an `age` property
// along with a method called `syncToDatabase()`. As a developer, I want the AI to
// increment the age property and sync that to a database. Note that `syncToDatabase` is an
// asynchronous operation, and we want any subsequent code to be aware of this.
// Also, name is a string and age is a number.

let person;

There's a big change there. Did you catch it?

- // consider a variable named `person`. Imagine it has a `name` and an `age` property
+ // Imagine `person` has a `name` and an `age` property
  // along with a method called `syncToDatabase()`. As a developer, I want the AI to
  // increment the age property and sync that to a database. Note that `syncToDatabase` is an
  // asynchronous operation, and we want any subsequent code to be aware of this.
  // Also, name is a string and age is a number.
  
+ let person;

Notice that we were able to incrementally add code prompts to our existing prompt. They can live side-by-side. It's not a completely different language, it's just a superset.

Plus, we really shaved off a lot of characters from our prompt. This is extra good for language models with small token windows. So let's keep going! Can we shorten the description of the properties and methods on person?

// As a developer, I want the AI to
// increment the age property and sync that to a database. Note that `syncToDatabase` is an
// asynchronous operation, and we want any subsequent code to be aware of this.
// Also, name is a string and age is a number.

let person: {
    name;
    age;
    syncToDatabase();
};

At this point, you might be satisfied, but we can go even further. You see, even our code prompt has information about the member names of person, we had to use lengthy comment prompts to describe those members. We can go deeper!

// As a developer, I want the AI to
// increment the age property and sync that to a database.

let person: {
    name: string;
    age: number;
    syncToDatabase(): Promise<void>;
};

This prompt is practically perfection. Who knows what the AI will do with the code prompts we've provided it? I don't, but I'm excited to find out!

person.age++;
person.syncToDatabase().then(() => {
    console.log('Synced to database');
});

Single quotes? Gross.

Fine-Grained Completions

Occasionally, inline completions might be a bit too aggressive. Occasionally, as you program, you might find it valuable to explicitly request completions on the next token. This is where fine-grained completions come in.

Rather than completing the entire line, you can request completions for just the next token. This is especially useful when you want to guide the AI to complete your code in a particular way. The UI for this is extremely lightweight.

This feature brings us the Intelligence of AI, but allows us to complete using our own discretion and common Sense.

Obviously, it should be named "SensiGence".

Checking for Validity

One of the key benefits of code prompts is that they can be used to check the validity of code completions. By providing a code prompt, you can ensure that the AI-generated code adheres to the constraints you've set. This is especially useful when you want to ensure that the AI doesn't generate code that violates your business logic or architectural patterns.

In short: what if something could check what the AI typed?

This would be a game changer. The biggest question is how we'll be able to train a model of this complexity. If I can raise $10 trillion to procure enough GPU power, then I think it's within reach. So feel free to sponsor me on GitHub!

[pgodman]

Approved. Begin implementation immediately, mortal.

[jits]

Yes, but, is it powered by The Blockchain??

[rubiesonthesky]

Just add tea.yml file to the repo and you will get the money in no time.

[DanielRosenwasser]

@jits see #48513

[jits]

@DanielRosenwasser — ahh yes, that was a classic!

[mrT23]

It does sound very reasonable that if you want to generate code, than your prompt will be more ... code-like 💪

just want to mention that someone really smart (not sure who) suggested using Pydantic class and its equivalence to YAML format to guide LLMs in code generation.
https://arxiv.org/pdf/2401.08500.pdf

This method has most of the advantages you mentioned above, and maybe some extra (or not ? ), since YAML format is very suitable for code generation.
This is not just an academic work, but actually used in practice, for example
https://github.com/Codium-ai/pr-agent/blob/main/pr_agent/settings/pr_code_suggestions_prompts.toml

[GorvGoyl]

petition to rename typescript to aiscript