Add model_picker toolset for dynamic model switching#1937
Add model_picker toolset for dynamic model switching#1937dgageot merged 1 commit intodocker:mainfrom
Conversation
Introduce a new 'model_picker' toolset that lets agents switch between LLM models mid-conversation. This enables agents to pick the best model for each sub-task (e.g. a fast model for simple questions, a powerful one for complex reasoning) and revert when done. Changes: - Add ModelPickerTool with change_model and revert_model tools - Register model_picker in the toolset registry - Wire runtime handlers (handleChangeModel, handleRevertModel) that validate the requested model against an allowed list, update the agent's model, and emit AgentInfo events so the UI reflects changes - Simplify registerDefaultTools to a plain handler map instead of pre-resolving tool definitions - Add config validation: 'models' field is required and exclusive to model_picker toolsets - Update agent-schema.json with model_picker type and models field - Add example configuration (examples/model_picker.yaml) - Add comprehensive unit tests for ModelPickerTool Signed-off-by: David Gageot <david.gageot@docker.com>
There was a problem hiding this comment.
Review Summary
Assessment: 🟡 NEEDS ATTENTION
The model_picker implementation is well-structured overall, but there's a validation gap that could allow invalid model configurations to pass through to runtime.
Found 1 issue requiring attention:
- 1 MEDIUM severity issue in config validation
Key Finding
The validation logic checks that the models list is non-empty for model_picker toolsets, but it doesn't validate the contents of that list. This means configurations with empty strings (e.g., models: [""]) would pass validation but cause runtime errors when the agent attempts to switch models.
Recommendation
Add validation to iterate over t.Models and ensure each entry is a non-empty string with valid model identifier format.
| return errors.New("openapi toolset requires a url to be set") | ||
| } | ||
| case "model_picker": | ||
| if len(t.Models) == 0 { |
There was a problem hiding this comment.
MEDIUM: Missing validation for individual model strings
The validation checks that len(t.Models) > 0, but it doesn't validate whether each string in the Models slice is non-empty or valid. A configuration like this would pass validation:
model_picker:
models:
- ""
- "gpt-4"But would cause runtime errors when the agent tries to switch to the empty model string. Consider adding:
for i, model := range t.Models {
if strings.TrimSpace(model) == "" {
return fmt.Errorf("toolset %q: models[%d] cannot be empty", t.Name, i)
}
}
Introduce a new 'model_picker' toolset that lets agents switch between LLM models mid-conversation. This enables agents to pick the best model for each sub-task (e.g. a fast model for simple questions, a powerful one for complex reasoning) and revert when done.
Changes: