-
Notifications
You must be signed in to change notification settings - Fork 10
Closed
Labels
enhancementNew feature or requestNew feature or requestpostponedThis issue/PR is postponed until there is a very good reason (e.g. $$$) to implement it.This issue/PR is postponed until there is a very good reason (e.g. $$$) to implement it.
Milestone
Description
- Introduce an "ID" method to the tool interface (like we have for model and provider) so the tools can be addressed deterministically instead of using the BinaryName method, which depends on the OS
- Allow to filter by tool IDs in command
install-toolsand add the following test for all OSes (currently we only test Linux there):
validate(t, &testCase{
Name: "Filtered",
Arguments: []string{"symflower"},
ExpectedInstalledToolNames: []string{
"symflower" + osutil.BinaryExtension(),
},
})
- Make Ollama version dependent. We want to use a minimum version like we do with Symflower. There is surely lot of code that we can share. (latest version is usually also faster!)
- how to integration test Ollama in the CI? is there a "dummy" model that always does the same thing?
- use random ports for testing to avoid the synchronization of a single Ollama instance
- run models that are not pulled yet
- query available models JSON list of available models ollama/ollama#3922
- download selected models before the evaluation starts
- better integration testing
- we currently just test with a small model that it does not error, but it would be nicer to have something deterministic Mocked model for integration testing Ollama into other tools and environments ollama/ollama#4196
- allow to customize the Ollama server port (and host?) and remove the workaround that restricts to running only one test (depending on Ollama) at a time
- comment why we have a wait delay in the exec util
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestpostponedThis issue/PR is postponed until there is a very good reason (e.g. $$$) to implement it.This issue/PR is postponed until there is a very good reason (e.g. $$$) to implement it.