Skip to content

Add request type muxing #96

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Mar 3, 2025
Merged

Add request type muxing #96

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
563b420
Add request type filtering to muxing docs
danbarr Mar 1, 2025
6d29a4e
Update changelog
danbarr Mar 1, 2025
22f46a4
Add precedence note
danbarr Mar 3, 2025
5302bb2
Update based on current behavior
danbarr Mar 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions docs/about/changelog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,10 @@ Major features and changes are noted here. To review all updates, see the

Related: [Upgrade CodeGate](../how-to/install.md#upgrade-codegate)

- **Request type muxing** - 26 Feb, 2025\
Workspace model muxing now supports filtering based on chat and FIM request
types. Check the [model muxing docs](../features/muxing.mdx) to learn more.

- **New integration: Open Interpreter** - 20 Feb, 2025\
CodeGate v0.1.24 adds support for the
[avante.nvim](https://github.com/yetone/avante.nvim) plugin for Neovim with
Expand Down
49 changes: 33 additions & 16 deletions docs/features/muxing.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ especially useful when you're working on multiple projects or tasks that require
different AI models.

In each of your CodeGate workspaces, you can select the AI provider and model
combinations to use, even dynamically switching the active model based on file
types found in your prompt. Then, configure your AI coding tool to use the
CodeGate muxing endpoint `http://localhost:8989/v1/mux` as an OpenAI-compatible
API provider.
combinations to use, even dynamically switching the active model based on the
request type and file types found in your prompt. Then, configure your AI coding
tool to use the CodeGate muxing endpoint `http://localhost:8989/v1/mux` as an
OpenAI-compatible API provider.

To change the model(s) currently in use, simply switch your active CodeGate
workspace.
Expand All @@ -38,9 +38,11 @@ flowchart LR
CodeGate --> WS1
CodeGate --> WS2
CodeGate --> WS3
WS1 --> |api| LLM1
WS2 --> |api| LLM2
WS3 --> |api| LLM3
WS1 --> |FIM requests| LLM1
WS1 --> |Chat| LLM2
WS2 --> |.md files| LLM2
WS2 --> |.js files| LLM3
WS3 --> |All prompts| LLM3
Comment on lines +41 to +45
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have one suggestion for .tsx and .ts files...actually if we have a .ts rule with an higher priority than .tsx we are going to match always the .ts rule.
So I would specify this in the doc

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fixed this part in the code, i.e. now it will correctly match .tsx and .ts

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh that's amazing @aponcedeleonch 🙌

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just added a note about precedence to the PR based on @peppescg's original comment, is that not correct now?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The note I added says:

Keep substring matching in mind. For example, if you have a filter for .ts files placed above a rule for .tsx files, the .tsx rule cannot be reached because the .ts filter matches first.

I think that's still accurate?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think my suggestion could be skipped now, cause @aponcedeleonch fixed in the code directly, sorry @danbarr 🙏

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, not accurate anymore since we're using a more explicit function to evaluate the substrings. We were using a simple in and now we use endswith. Check the following python output

>>> file = "test.tsx"
>>> ".ts" in file
True
>>> file.endswith(".ts")
False

Copy link
Contributor Author

@danbarr danbarr Mar 3, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK I just updated again based on our conversation, I think this is now accurate to the current behavior: 5302bb2

```

## Use cases
Expand Down Expand Up @@ -109,21 +111,33 @@ workspace selection menu or the
In the **Model Muxing** section, select the default ("catch-all") model to use
with the workspace.

To assign a different model based on filename, click **Add Filter**. In the
**Filter by** column, enter a file name or extension string to match. This is a
simple substring match, wildcards are not supported. For example, to match
Python files, enter `.py`. Then select the model to use with that file type.
To assign a different model based on request type or filename, click **Add
Filter**.

In the **Request Type** column, select the type of prompt to match:

- `FIM & Chat` matches all prompt types
- `FIM` matches fill-in-the-middle (completion) requests
- `Chat` matches chat prompts

In the **Filter by** column, enter a file name or extension string to match.
This is an exact or `endswith` match, wildcards are not supported. For example,
to match Python files, enter `.py`.

Finally, select the model to use for prompts that match the rule.

Filter rules are evaluated top-down. CodeGate selects the active model for a
request using the first matching rule. If the prompt contains multiple files in
context, the first rule that matches _any_ of the files is used. If no filter is
matched, the catch-all rule applies.

#### Example configuration

<ThemedImage
alt='An example showing several muxing rules for different file types'
sources={{
light: useBaseUrl('/img/features/muxing-rules-light.webp'),
dark: useBaseUrl('/img/quickstart/muxing-rules-dark.webp'),
dark: useBaseUrl('/img/features/muxing-rules-dark.webp'),
}}
title='Example muxing rules'
width={'800px'}
Expand All @@ -132,10 +146,13 @@ _An example showing several muxing rules for different file types_

Breaking down the above example:

- Markdown files (`.md`) use the gpt-4o-mini model from the OpenAI provider.
- JavaScript and TypeScript files (`.js` and `.ts`, which also matches `.jsx`
and `.tsx`) use anthropic/claude-3.5-sonnet via OpenRouter.
- All other requests use Ollama.
- Markdown files (`.md`) use the gpt-4o-mini model from the OpenAI provider for
all request types.
- Chat prompts for JavaScript files (`.js`, but NOT `.jsx`) use
anthropic/claude-3.7-sonnet via OpenRouter.
- FIM requests for all files (except `.md` since it's higher in the list) use
Ollama with the qwen2.5-coder:1.5b model.
- All other requests use anthropic/claude-3.5-sonnet via OpenRouter.
- A request containing both a JavaScript and Markdown file will match the `.md`
rule first and use OpenAI.

Expand Down
Binary file modified static/img/features/muxing-rules-dark.webp
View file
Open in desktop
Binary file not shown.
Binary file modified static/img/features/muxing-rules-light.webp
View file
Open in desktop
Binary file not shown.