Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v1.4 refactor and rename #31

Merged
merged 36 commits into from
Jan 6, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
bef173b
Add ESLint configuration and Prettier settings; update VSCode configu…
codybrom Jan 6, 2025
48b4fd3
Merge remote-tracking branch 'origin/main' into v4
codybrom Jan 6, 2025
a5c34bb
Fix formatting in README for clarity and consistency
codybrom Jan 6, 2025
c84f2d9
Update pretest script to use 'build' instead of 'compile'
codybrom Jan 6, 2025
b298eaa
Remove deprecated ESLint configuration file and update ignore pattern…
codybrom Jan 6, 2025
2752ef7
Add TypeScript SDK path to VSCode settings
codybrom Jan 6, 2025
15f6089
Update ESLint script to remove file extension specification
codybrom Jan 6, 2025
5981648
Refactor ignore module import and update variable declarations for co…
codybrom Jan 6, 2025
ad52af3
Remove unused test files and related test runner code
codybrom Jan 6, 2025
a9af97c
Migrate Prettier configuration from .prettierrc.json to package.json
codybrom Jan 6, 2025
54cf597
Remove Airbnb code style badge from README
codybrom Jan 6, 2025
acca191
Add custom spell check words to VSCode settings
codybrom Jan 6, 2025
888b2f1
Add utility functions for token estimation, import extraction, and fi…
codybrom Jan 6, 2025
764b621
Add command handler and context generator for GPT-friendly context cr…
codybrom Jan 6, 2025
0fdc582
Remove Prettier configuration path from VSCode settings
codybrom Jan 6, 2025
4153742
Refactor file utility functions and reorganize file structure
codybrom Jan 6, 2025
a3da8fc
Rename treeProvider to markedFilesProvider and update file path utili…
codybrom Jan 6, 2025
05c64f5
move importParser and markdownUtils into utils
codybrom Jan 6, 2025
518b108
Replace gpt-3-encoder with @dqbd/tiktoken, add highlight.js, and refa…
codybrom Jan 6, 2025
395a161
move commandHandler and contextGenerator files, and update import pat…
codybrom Jan 6, 2025
5f7bc7a
Refactor command handling: remove CommandHandler, implement clear and…
codybrom Jan 6, 2025
ce3ea97
Refactor context and marked files handling: streamline imports and en…
codybrom Jan 6, 2025
b4529ca
Refactor IgnoreManager: convert to instance-based implementation and …
codybrom Jan 6, 2025
ea8dc05
Refactor formatFileComment: simplify return value to include language…
codybrom Jan 6, 2025
60aada8
Add SVG icons for clear actions and enhance marked files functionality
codybrom Jan 6, 2025
6d5f8be
Refactor MarkedFilesProvider: streamline file watcher initialization …
codybrom Jan 6, 2025
b195ae7
Refactor markFile functionality: unify marking logic for files and di…
codybrom Jan 6, 2025
0bc41a4
Refactor markFile command: improve error handling, streamline file ma…
codybrom Jan 6, 2025
4a5ac4c
chore: update VSCode engine version to ^1.96.0
codybrom Jan 6, 2025
e08ff95
chore: optimize build process with esbuild
codybrom Jan 6, 2025
711a1e8
release: update version to 1.4.0 in package.json
codybrom Jan 6, 2025
4f2fa19
chore: remove unused prettier configuration from VSCode settings
codybrom Jan 6, 2025
8ecdc4d
Refactor: update project name and description to reflect LLM context …
codybrom Jan 6, 2025
dae36fa
feat: add token warning threshold configuration and update package.js…
codybrom Jan 6, 2025
a02893e
feat: add support for custom ignore files in context generation
codybrom Jan 6, 2025
4d143c1
docs: update README to add ignore files section
codybrom Jan 6, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Refactor: update project name and description to reflect LLM context …
…generation; modify clipboard success message and file comment formatting
  • Loading branch information
codybrom committed Jan 6, 2025
commit 8ecdc4d1bfba32410d6d98c1377c1e6d0b04fcd6
2 changes: 1 addition & 1 deletion .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Dependabot configuration for GPT Context Generator VSCode plugin
# Dependabot configuration for LLM Context Generator VSCode plugin

version: 2
updates:
Expand Down
107 changes: 61 additions & 46 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,71 +1,86 @@
# GPT Context Generator
# LLM Context Generator

[![VS Marketplace Version](https://badgen.net/vs-marketplace/v/codybrom.gpt-context-generator)](https://marketplace.visualstudio.com/items?itemName=codybrom.gpt-context-generator)
[![VS Marketplace Installs](https://badgen.net/vs-marketplace/i/codybrom.gpt-context-generator)](https://marketplace.visualstudio.com/items?itemName=codybrom.gpt-context-generator)
[![VS Marketplace Rating](https://badgen.net/vs-marketplace/d/codybrom.gpt-context-generator)](https://marketplace.visualstudio.com/items?itemName=codybrom.gpt-context-generator)

This Visual Studio Code extension helps you generate context for .js/.jsx and .ts/.tsx files, making
it easier to collaborate with AI models like OpenAI's GPT-4. The extension generates context by
pulling in local dependencies such as API pages that are referenced in your code. It also respects
`.gitignore` rules to avoid including unnecessary files.
This Visual Studio Code extension helps you generate LLM-ready context from your workspace files, making it easier to collaborate with AI models. The extension can automatically reference local code dependencies and respects your `.gitignore` rules to avoid including unnecessary files.

## Features

- Generate GPT-compatible multi-file context from the currently open file and local imports
- Generate GPT-compatible multi-file context of the entire VS Code workspace
- Generate GPT-compatible multi-file context for marked files
- Mark/Unmark files for inclusion in generated context
- Estimate the number of OpenAI tokens in the generated context
- Generate LLM-ready context from the currently open file and its imports, your entire VS Code workspace or marked files
- Mark or unmark open files via Command Palette, or individual files and folders via the Explorer context menu
- Automatic file tracking updates marked files when they're moved or deleted
- Token count estimation for generated context
- Automatic support for multiple programming languages and file types

## Usage

### Generate context for the currently open file and its local imports
### Mark Files for Context

1. Open a file in Visual Studio Code.
2. Press `Ctrl+Shift+P` (Windows) or `Cmd+Shift+P` (Mac) to open the Command Palette.
3. Type `Generate GPT Context (Current File + Imports)` and select the command from the list.
4. The generated context, including dependencies, will be displayed in a new editor tab or copied
to the clipboard based on your configuration.
1. **Via Explorer**:
- Right-click on one or more files or folders in the Explorer
- Select "Mark for LLM Context"
- Files appear in the Marked Files view
- Marking a folder includes all compatible files within it

### Generate context for the entire workspace
2. **Via Command Palette**:
- Open a file
- Press `Ctrl+Shift+P` (Windows) or `Cmd+Shift+P` (Mac)
- Select "Mark/Unmark File for LLM Context"

1. Open a workspace in Visual Studio Code.
2. Press `Ctrl+Shift+P` (Windows) or `Cmd+Shift+P` (Mac) to open the Command Palette.
3. Type `Generate GPT Context (Workspace)` and select the command from the list.
4. The generated context will be displayed in a new editor tab or copied to the clipboard based on
your configuration.
### Generate Context

### Generate context for marked files
Three ways to generate context:

1. Open a file in Visual Studio Code.
2. Press `Ctrl+Shift+P` (Windows) or `Cmd+Shift+P` (Mac) to open the Command Palette.
3. Type `Mark/Unmark File for Inclusion` and select the command to mark the file for inclusion.
4. Repeat steps 1-3 for all files you want to include in the context.
5. Press `Ctrl+Shift+P` (Windows) or `Cmd+Shift+P` (Mac) to open the Command Palette.
6. Type `Generate GPT Context (Marked Files)` and select the command from the list.
7. The generated context for marked files will be displayed in a new editor tab or copied to the
clipboard based on your configuration.
1. **Current File + Imports**:
- Open a file
- Command Palette → "Generate LLM Context (Current File + Imports)"

2. **Workspace**:
- Open a workspace
- Command Palette → "Generate LLM Context (Workspace)"

3. **Marked Files**:
- Mark desired files
- Command Palette → "Generate LLM Context (Marked Files)"

The generated context will be copied to your clipboard or opened in a new window, based on your settings.

## Token Count Estimation

When generating context, the extension will also display an information message with an estimated
number of OpenAI tokens in the generated text. This information can be helpful when working with AI
models that have token limitations.
After generating context, you'll see an estimated token count. This helps you stay within AI model token limits. A warning appears if the context exceeds 8,000 tokens.

## Configuration

From settings, you can configure the extension to work as you prefer.

- **Output method**
- Copy to Clipboard (_default_)
- New Window
- **Output language** (only applies when 'outputMethod' is set to 'newWindow')
- Plaintext (_default_)
- Markdown
- **Include package.json in open-file context** (_default: true_)
- **Set File Extensions to Include in Workspace Context**
- _defaults: js, jsx, ts, tsx, mdx, json_
Configure the extension in VS Code settings:

- **Output Method**
- `clipboard`: Copy to clipboard (default)
- `newWindow`: Open in new editor

- **Output Language** (for newWindow only)
- `plaintext`: Plain text (default)
- `markdown`: Markdown formatting

- **Include package.json**
- Include package.json when generating context for open file
- Default: `true`

- **Detected File Extensions**
- Customize which file types to include
- Supports many languages and formats:
- JavaScript/TypeScript (js, jsx, ts, tsx, etc.)
- Python (py, pyi, pyw, ipynb)
- Ruby (rb, rake, erb, etc.)
- PHP (php, phtml)
- Swift/Objective-C (swift, m, h, etc.)
- Systems (c, cpp, rs, go, etc.)
- Web (html, css, scss, etc.)
- Mobile (java, kt, dart, etc.)
- Configuration (json, yaml, toml, etc.)
- And more...

## License

This extension is released under the MIT License.
Released under the MIT License.
127 changes: 110 additions & 17 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "gpt-context-generator",
"displayName": "GPT Context Generator",
"description": "Generate GPT-friendly context for files or workspaces",
"displayName": "LLM Context Generator",
"description": "Generate LLM-ready context for files or workspaces",
"version": "1.4.0",
"engines": {
"vscode": "^1.96.0"
Expand All @@ -28,19 +28,19 @@
"commands": [
{
"command": "gpt-context-generator.createGPTFriendlyContextForOpenFile",
"title": "Generate GPT Context (Current File + Imports)"
"title": "Generate LLM Context (Current File + Imports)"
},
{
"command": "gpt-context-generator.createGPTFriendlyContext",
"title": "Generate GPT Context (Workspace)"
"title": "Generate LLM Context (Workspace)"
},
{
"command": "gpt-context-generator.markFileForInclusion",
"title": "Mark/Unmark File for Inclusion"
"title": "Mark/Unmark File for LLM Context"
},
{
"command": "gpt-context-generator.createGPTFriendlyContextForMarkedFiles",
"title": "Generate GPT Context (Marked Files)"
"title": "Generate LLM Context (Marked Files)"
},
{
"command": "gpt-context-generator.clearMarkedFiles",
Expand All @@ -60,26 +60,124 @@
},
{
"command": "gpt-context-generator.markFilesFromExplorer",
"title": "Mark for GPT Context"
"title": "Mark for LLM Context"
}
],
"configuration": {
"title": "GPT Context Generator",
"title": "LLM Context Generator",
"properties": {
"gpt-context-generator.detectedFileExtensions": {
"type": "array",
"items": {
"type": "string"
"type": "string",
"enum": [
"js",
"jsx",
"ts",
"tsx",
"mjs",
"cjs",
"vue",
"svelte",
"astro",
"njk",
"liquid",
"py",
"pyi",
"pyw",
"ipynb",
"rb",
"rake",
"gemspec",
"erb",
"slim",
"haml",
"php",
"phtml",
"swift",
"m",
"mm",
"h",
"hpp",
"c",
"cpp",
"cc",
"cxx",
"rs",
"go",
"zig",
"html",
"css",
"scss",
"sass",
"less",
"java",
"kt",
"kts",
"gradle",
"plist",
"dart",
"xaml",
"lua",
"sh",
"bash",
"zsh",
"bat",
"cmd",
"nsh",
"json",
"yaml",
"yml",
"toml",
"ini",
"env",
"xcconfig",
"podspec",
"editorconfig",
"dockerignore",
"gitignore",
"eslintrc",
"prettierrc",
"stylelintrc",
"babelrc",
"makefile",
"dockerfile",
"k8s",
"md",
"mdx",
"txt",
"tex",
"svg",
"sql",
"prisma",
"csv",
"tsv",
"xml"
]
},
"default": [
"js",
"jsx",
"ts",
"tsx",
"mdx",
"json"
"py",
"rb",
"php",
"swift",
"java",
"kt",
"cs",
"html",
"css",
"json",
"md",
"txt",
"yml",
"env"
],
"description": "File extensions to be detected and included in the generated context."
"description": "File extensions that will be included when generating context. Supports multiple programming languages, frameworks, tools, and development workflows.",
"markdownDescription": "File extensions that will be included when generating context. Supports:\n\n- **JavaScript/TypeScript**: js, jsx, ts, tsx, mjs, cjs, vue, svelte, astro, njk, liquid\n- **Python**: py, pyi, pyw, ipynb\n- **Ruby**: rb, rake, gemspec, erb, slim, haml\n- **PHP**: php, phtml\n- **Swift & Objective-C**: swift, m, mm, h, hpp\n- **Systems Programming**: c, cpp, cc, cxx, rs, go, zig\n- **Web/Styling**: html, css, scss, sass, less\n- **Mobile & Cross-Platform**: java, kt, kts, gradle, plist, dart, xaml\n- **Shell & Scripts**: sh, bash, zsh, bat, cmd, nsh\n- **Data/Config**: json, yaml, yml, toml, ini, env, xcconfig, podspec, editorconfig, dockerignore, gitignore, eslintrc, prettierrc, stylelintrc, babelrc\n- **Build & Deploy**: makefile, dockerfile, k8s\n- **Documentation**: md, mdx, txt, tex\n- **Graphics**: svg\n- **Query Languages**: sql, prisma\n- **Data Formats**: csv, tsv, xml",
"order": 1
},
"gpt-context-generator.outputMethod": {
"type": "string",
Expand All @@ -104,11 +202,6 @@
"type": "boolean",
"default": true,
"description": "Include package.json as part of the context (Open File Only)."
},
"gpt-context-generator.fileCommentFormat": {
"type": "string",
"default": "{filePath}\\n```{markdownLang}\\n{fileContent}\\n```",
"description": "The format of the file comment in the generated context. Use {filePath}, {markdownLang}, and {fileContent} as placeholders."
}
}
},
Expand Down
10 changes: 3 additions & 7 deletions src/generators/contextGenerator.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,14 +25,12 @@ import {
export class ContextGenerator {
private config: WorkspaceConfiguration;
private detectedFileExtensions: string[];
private format: string;

constructor(private workspacePath: string) {
this.config = workspace.getConfiguration('gpt-context-generator');
this.detectedFileExtensions = this.config.get(
'detectedFileExtensions',
) as string[];
this.format = this.config.get('fileCommentFormat') as string;
initializeIgnoreFilter(workspacePath);
}

Expand All @@ -58,9 +56,7 @@ export class ContextGenerator {
await window.showTextDocument(document, ViewColumn.One);
} else if (outputMethod === 'clipboard') {
await env.clipboard.writeText(content);
window.showInformationMessage(
'GPT-friendly context copied to clipboard.',
);
window.showInformationMessage('LLM-ready context copied to clipboard.');
}
}

Expand Down Expand Up @@ -201,7 +197,7 @@ export class ContextGenerator {
const fileExtension = getExtension(filePath);
if (this.detectedFileExtensions.includes(fileExtension)) {
const fileData = this.createFileData(filePath, relPath);
contextParts.push(`${formatFileComment(this.format, fileData)}\n\n`);
contextParts.push(`${formatFileComment(fileData)}\n\n`);
}
}

Expand All @@ -213,7 +209,7 @@ export class ContextGenerator {
extension: 'json',
content,
};
contextParts.push(`${formatFileComment(this.format, fileData)}\n\n`);
contextParts.push(`${formatFileComment(fileData)}\n\n`);
}
}
}
Expand Down
13 changes: 4 additions & 9 deletions src/utils/markdownUtils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,8 @@ export const getMarkdownLang = (extension: string): string => {
return language?.name ?? ext;
};

export const formatFileComment = (format: string, file: FileData): string => {
const replacements = {
filePath: file.path,
markdownLang: `${getMarkdownLang(file.extension)} ${file.path}`,
fileContent: file.content,
};

// Return just the code block without the filename header
return `\`\`\`${replacements.markdownLang}\n${replacements.fileContent}\`\`\``;
export const formatFileComment = (file: FileData): string => {
const markdownLang = getMarkdownLang(file.extension);
// Always use the standard format: filename in language specifier
return `\`\`\`${markdownLang} ${file.path}\n${file.content}\`\`\``;
};
3 changes: 1 addition & 2 deletions src/utils/vscodeUtils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,7 @@ export const showMessage = {
info: (message: string) => window.showInformationMessage(message),
error: (message: string) => window.showErrorMessage(message),
warning: (message: string) => window.showWarningMessage(message),
copySuccess: () =>
showMessage.info('GPT-friendly context copied to clipboard.'),
copySuccess: () => showMessage.info('LLM-ready context copied to clipboard.'),
tokenCount: (count: number) => {
const message = `The generated context is approximately ${count} tokens${
count > 8000 ? ', which is greater than 8000 tokens' : ''
Expand Down
Loading