feat: add URL backend for custom LLM endpoints#9640
Conversation
Ali-Sab
force-pushed
the
feat/add-llm-url-option
branch
2 times, most recently
from
786adcb to
2e974e2
Compare
|
|
||
| export const ACCEPTED_NODE_CA_FILE_EXTS = ['.pem', '.crt', '.cer', '.p12']; | ||
|
|
||
| export const LLM_BACKENDS = ['gguf', 'claude', 'openai', 'gemini'] as const; |
There was a problem hiding this comment.
https://github.com/Kong/insomnia/blob/develop/packages/insomnia/src/plugins/types.ts#L3-L25 is used in a few places and is missing the new backend type, along with the url field. Right now, the generate commit diffs and mcp sampling both use this interface and would likely fail if the url backend were configured
There was a problem hiding this comment.
Good catch. Let me update this, and some other fields in this file to show what the LLM URL option can support
| try { | ||
| setIsLoadingModels(true); | ||
| setError(null); | ||
| const response = await fetch(`${realUrl}/models`); |
There was a problem hiding this comment.
Should we be wrapping this in a timeout for this or some sort of option to prevent it from hanging if it's incorrect?
There was a problem hiding this comment.
Should we be using new URL to get better path safety?
There was a problem hiding this comment.
adding both of these in the next revision, let me know your thoughts on them 🙏
Ali-Sab
force-pushed
the
feat/add-llm-url-option
branch
2 times, most recently
from
da322ea to
f12fb62
Compare
|
Additional changes since initial commit:
|
| // Enforce minimum maxTokens to prevent unreasonably low limits from MCP servers | ||
| // that would result in truncated/incomplete responses and silent failures. | ||
| const MIN_MAX_TOKENS = 1000; | ||
| if (finalModelConfig.maxTokens !== undefined && finalModelConfig.maxTokens < MIN_MAX_TOKENS) { |
There was a problem hiding this comment.
Should we instead be surfacing an error or enforcing this in the settings so this behaviour is more visible?
There was a problem hiding this comment.
Ideally, yes. I'd rather not have some automatically enforced value and instead give the user the ability to specify a max token override.
I can try adding something of that sort, like a Max Token Override in the settings. We should ideally have this for all types of AI models, but it felt like this PR might get too big. Thoughts going forward?
There was a problem hiding this comment.
Perhaps we should add this minimum max token enforcement as part of a follow-up in that case?
There was a problem hiding this comment.
I'd be okay with that, and we can try a better quality solution with it. I'll make a rev to remove this out for now
* Add support for configuring custom OpenAI-compatible LLM endpoints via URL. * Includes new URL backend component, backend service updates, unit tests, and * UI consistency improvements across all LLM configuration components.
Ali-Sab
force-pushed
the
feat/add-llm-url-option
branch
from
f12fb62 to
298b15d
Compare
* Add support for configuring custom OpenAI-compatible LLM endpoints via URL. * Includes new URL backend component, backend service updates, unit tests, and * UI consistency improvements across all LLM configuration components.
3 participants
Feature: Custom URL Backend
Adds support for OpenAI-compatible LLM endpoints via custom URLs.
Backend
urlandbaseURLto theLLMConfiginterface.LLM_BACKENDSconstant.Frontend
aiFeaturesobject.Consistency Updates
Standardized the UI across Claude, Gemini, OpenAI, and GGUF:
Files Changed
constants.ts(Added URL backend type)llm-config-service.ts&.test.ts(Backend logic and tests)ai-settings.tsx&url.tsx(Settings UI and new component)claude.tsx,gemini.tsx,openai.tsx,gguf.tsx(UI polish)