How do I change the model?

[bemorecyber-rab]

Hi all,

Love this and want to get using it. I am using a Windows machine and got fabric installed using WSL + VS Code.

Went to use it and it says I need to use GPT-4 ... which is fine, I don't pay for it (yet!) but wanted to test this with the older models.
I saw the fabric -m MODEL command but when I run it, it just hangs.. or I have to fail it.

Has anyone else downgraded the model being used?

I can list models fine "fabric --listmodels" but none of them seems to work.

[ben0815]

pbpaste|fabric -m gpt-3.5-turbo -p extract_wisdom
works for me. It's the exact name from the listmodels command.

[bemorecyber-rab]

ahhh you are calling the model with the prompt ... I thought I could just change the model as the default - this makes sense

So I need to call the model and then call the pattern - makes sense. Thanks @ben0815

[nic0711]

It would be great if we could also use local Mistral-7b... ;)

[xntrik]

Even better, you can just set your env vars with OPENAI_BASE_URL=http://localhost:11434/v1

This gets picked up by the OpenAI Py library, and will "just work".

For example, if I edit my ~/.config/fabric/.env to look like:

OPENAI_API_KEY=llama
OPENAI_BASE_URL=http://localhost:11434/v1

Then this will work.

The --listmodels command won't work though, and I don't believe this supports streaming either.

[nic0711]

Yeah!
It works with LM Studio too...

OPENAI_API_KEY=lmstudio
OPENAI_BASE_URL=http://localhost:1234/v1

Of course you have to adjust the local port.

Start LM Studio, choose your model, start the server - have fun.

[zestysoft]

Seems like I'm missing a step somewhere?

$ cat ~/.config/fabric/.env 
OPENAI_API_KEY=lmstudio
OPENAI_BASE_URL=http://localhost:8000/v1

$ echo "An idea that coding is like speaking with rules." | fabric --model mistral:instruct --pattern write_essay
Failed to fetch models: HTTP 401

EDIT:

I had to give it a real openai api key and re-run fabric --setup. Afterwards, I had to install ollama, and run a model in order for fabric --listmodels to show a local model. Finally I could supply the OPENAI_BASE_URL setting above and have it properly access the locally hosted model making sure to supply the model in the command line.

[nic0711]

For me it works without an OpenAI key - just start the local server with local model.

Code 401: "Unauthorized"

[zestysoft]

Strange. What happens if you run fabric --listmodels?

If I have a valid OPENAI_API_KEY, I get this output:

GPT Models:
gpt-3.5-turbo
gpt-3.5-turbo-0125
gpt-3.5-turbo-0301
gpt-3.5-turbo-0613
gpt-3.5-turbo-1106
gpt-3.5-turbo-16k
gpt-3.5-turbo-16k-0613
gpt-3.5-turbo-instruct
gpt-3.5-turbo-instruct-0914
gpt-4
gpt-4-0125-preview
gpt-4-0613
gpt-4-1106-preview
gpt-4-turbo-preview
gpt-4-vision-preview

Local Models:
mixtr

Claude Models:
claude-3-opus-20240229

However the moment I change the OPENAI_API_KEY value to something else like 'lmstudio', I again get that 'Failed to fetch models: HTTP 401' error.

Any chance you have that environment variable being set elsewhere? Like from the terminal if you run export|grep OPENAI nothing shows up?

EDIT

It's hard coded here:

fabric/installer/client/cli/utils.py

Line 260 in 24e1616

"https://api.openai.com/v1/models", headers=headers)

Fix here: #189

[xntrik]

I've got a small PR to allow the default model to be set outside of cli flags: #156

[xssdoctor]

It’s already added. —changeDefaultModel.

[xntrik]

Yass! Thank you @xssdoctor

[mgavish]

does this work for you?
yt --transcript https://youtube.com/watch?v=uXs-zPc63kM | fabric --stream --pattern extract_wisdom