First of all, a wonderful tool, it works great in getting what i want in a good structure.
I'm using a local llama.cpp docker server on the same computer. For some reason youve kept the 'local llama' to be 'localhost' only, but that doesn't get accepted. (also, you cant enter an api key in that field)
So i am using the 'custom api' and that works great.
It would be great if you could add an 'unload model' call to the same server (and apikey). For llama cpp that is http://localhost:8080/models/unload (notice it is without the /v1) as that saves people from manually needing to login to llama and unload the model (or set the sleep timeout extremely low)
implementation wise, it could be an input field where people can submit the 'unload' call, so it is flexible for different toolings.
And then in the generation tab a button to unload, or even a checkbox 'auto unload model after generation'
First of all, a wonderful tool, it works great in getting what i want in a good structure.
I'm using a local llama.cpp docker server on the same computer. For some reason youve kept the 'local llama' to be 'localhost' only, but that doesn't get accepted. (also, you cant enter an api key in that field)
So i am using the 'custom api' and that works great.
It would be great if you could add an 'unload model' call to the same server (and apikey). For llama cpp that is
http://localhost:8080/models/unload(notice it is without the/v1) as that saves people from manually needing to login to llama and unload the model (or set the sleep timeout extremely low)implementation wise, it could be an input field where people can submit the 'unload' call, so it is flexible for different toolings.
And then in the generation tab a button to unload, or even a checkbox 'auto unload model after generation'