Feedback about local LLM usage with paper-qa (share your experience about different LLMs and their parametrization).

[Snikch63200]

After a lot of trials with different local LLMs and different parameters (also called 'options'), I decided to open this discussion to share experiences about this.

My aim is to compare performances of different local models, with different configurations.

This is not a discussion about configuration issues but configuration optimization.
This is not, stricto sensu, a discussion about paper-qa settings

For each trial thanks to precise :

• Hardware (GPU type, CUDA CORES, VRAM, CPU type, CPU CORES, RAM)
• LLM serving type (Ollama, Llamafile, etc...)
• LLM model (llama3.1:70b, mixtral:8x7b, phi, etc...)
• Example (code snippet) of your configuration. Comments appreciated.
• Prompt used (with approximate tokens)
• PaperQA settings when not 'default', especially answer_max_sources, evidence_k and max_concurrent_requests that have a major impact on speed and answer relevance.
• A comment about answer quality.
• Request time.

New ideas for standardization of tests are welcome.

[Snikch63200]

As an example :

• Hardware : RTX 4090, 16 384 CUDA cores, 24 GB VRAM DDR6X, IntelCore I9-13900KF 3000 Mhz 24 cores 32 threads, 64 GB RAM DDR4
• serving type : ollama
• model : llama3.1:70b
• config :

local_llm_config = dict(
                        model_list=[
                                    dict(
                                        model_name=f"ollama/llama3.1:70b",
                                        litellm_params=dict(
                                                            model=f"ollama/llama3.1:70b",
                                                            api_type= "ollama",
                                                            api_base="http://192.168.1.15:11434",
                                                            cache= True,
                                                            mirostat=0,
                                                            mirostat_eta=0.1,
                                                            mirostat_tau=5.0,
                                                            num_ctx=32768, #Default 2048
                                                            repeat_last_n=64, # Sets how far back for the model to look back to prevent repetition. (Default: 64, 0 = disabled, -1 = num_ctx)
                                                            repeat_penalty=1.1,
                                                            temperature=0.2, #Low temperature to prevent consistency problems-default 0.7-
                                                            seed=0,
                                                            tfs_z=1,
                                                            num_predict=1000000,# Maximum number of tokens to predict when generating text. (Default: 128, -1 = infinite generation, -2 = fill context)
                                                            top_k=20, #Low top_k to prevent consistency problems-default 40-
                                                            top_p=0.9,
                                                            min_p=0.0
                                                            ),
                                        ),

                                    ]
                        )

• prompt "Quels sont les sites ayant livré des sépultures d'enfants, nouveaux nés ou périnataux enterrés dans des récipients en céramique (terrines, bassines, amphores, dolia). Répondez en Français." (about roman funeral archaelogy, in french - chose this question because it's very specific)
• answer_max_sources : 7, evidence_k : 20, max_concurrent_requests : 1
• The answer is globally relevant except 2 citations on 7 that concern ceramics buried near the corpse, not corpses buried inside ceramics -> LLM understanding problem.
• Request time : 0:35:56.063536

[maykcaldas]

Thanks for starting this discussion, @Snikch63200. It is very needed. PaperQA is highly customizable, and I noticed some users had trouble using the Settings class. Sometimes, some defaults were being used mistakenly.

So, I wrote this tutorial. I hope this will help users using paper-qa

[rosegarden-coder]

@maykcaldas It would be great if the tutorial also include an easy export of local LLMs, especially ollama, rather than only OPENA and ANTHROPIC api, as described in #899.

@Snikch63200 I borrowed your settings and it has been useful. I use ollama:

        llm="ollama/llama3.3:latest",  
        summary_llm="ollama/llama3.3:latest",
        embedding="ollama/granite3-dense:8b",

And the result is quite solid and accurate. Although occasionally I would like an exact quotation from the paper rather than an answer, but I do not know how to do it.

It is a llama3.3 70b model running on a total vram of 60GB (48+12). For my limited tests, It took approximately 2-5 minutes to process and get answers from less than 10 papers.

[maykcaldas]

Hey @rosegarden-coder.

Yeah, I saw your discussion and agree that setting up local llms deserves some attention. I don't use local llms often, though.

But please notice that the verbosity to set ollama comes from litellm. For instance, setting every llm and its llm_config is also required for anthropic. I understand that setting a longer command line is frustrating because ollama also requires passing the api_base inside litellm_params.

A momentary solution would be to use a script to invoke ask or to save your configuration so that you can invoke it with pqa -s my_config ask "What is...".
Check this doc.

Does it work for you?

[rosegarden-coder]

Many people can not use remote api due to privacy, cost or other reasons. And as I am getting solid results using local LLMs, I see no reason to use those apis.

My impression is that remote api are rooted in the paper-qa script and I read many users can not natively use local LLMs even they configured the setting.

Using -s is a good idea and I am particularly interested in https://github.com/Future-House/paper-qa/blob/main/paperqa/configs/contracrow.json. I am not using my desktop right now so I will test it later.

If I swap all the remote apis with local models in the contracrow file, will it work? Can you suggest the edits to make contracrow work for ollama?

[Snikch63200]

Thanks for your anwers @maykcaldas and @rosegarden-coder,

Cannot use @maykcaldas config because it reaches my hardware limits (without reaching offload limits)... That's strange because it perfectly works with open-webui. But Llama3.3+granite3-dense:8b seems a very promising combination.

As @rosegarden-coder explained, Local LLM's should get special attention.

[maykcaldas]

Hey @Snikch63200 and @rosegarden-coder

Can you please help me understand better what the problem is with Ollama?
I gave it a try today. I went to the ollama website, downloaded the server (I'm on a macos), started it and got the model by running ollama run llama3.2 on my terminal. Once the model was downloaded and the server was running, I could simply run this script:

import pathlib
from paperqa.settings import Settings, AgentSettings, IndexSettings

llm = "ollama/llama3.2"
llm_config = {
    "model_list": [
        {
            "model_name": llm,
            "litellm_params": {
                "model": llm,
                "api_base": "http://localhost:11434",
            },
        }
    ],
}

settings = Settings(
    llm=llm,
    llm_config=llm_config,
    summary_llm=llm,
    summary_llm_config=llm_config,
    agent=AgentSettings(
        agent_llm=llm,
        agent_llm_config=llm_config,
        index=IndexSettings(
            paper_directory=pathlib.Path.cwd().joinpath("papers"),
            index_directory=pathlib.Path.cwd().joinpath("papers/index"),
        ),
    ),
    embedding="st-multi-qa-MiniLM-L6-cos-v1",
)

from paperqa import ask

response = ask(
    "What are the most relevant language models used for chemistry?", settings=settings
)

And it runs normally.
Costs are not computed because litellm doesn't have ollama/llama3.2 model in their cost table (which prints this warning: Could not find cost for model ollama/llama3.2.).

I ran a test in which I killed the server while paper-qa was running. I could certify that it was using ollama/llama3.2 because I get this error if the server is not running.

Encountered exception during tool call for tool gather_evidence: litellm.APIConnectionError: OllamaException - All connection attempts failed. Received Model Group=ollama/llama3.2
Available Model Group Fallbacks=None LiteLLM Retried: 2 times, LiteLLM Max Retries: 3
Could not find cost for model ollama/llama3.2.

Do you have any suggestions for a different, simpler interface for that? The configuration required to run ollama is the same as that required to use any provider other than OpenAI.

[rosegarden-coder]

@maykcaldas

I am happy to see you get it running. If you could also set an ollama embedding model in your script, then you are completely running on ollama.

I will later show with console outputs on my desktop so you could have an better idea. Momentarily, please allow me explain:

First, When you say "The configuration required to run ollama is the same as that required to use any provider other than OpenAI", well, it is only true when you run it in the script as you have demonstrated. However, one can not run the pqa command with ollama as backend without some setting Kungfu -- and even that it may not work -- this manifests the importance of a simple environment variable export for ollama to make it work in pqa command.

Second, about Bundled Settings in contracrow.json. One can not run ollama with contracrow.json setting file directly without editing it. And even after setting the ollama/LocalModels in the config file, it may not work according to my limited previous tests. Can you also test it with pqa -s 'contracrow' to see if it is going to work? As a side note, since we may have already export the api variables (be it openai or claude or ollama), these config files should be api agnostic, and the user do not need to set the apis or LLMs to use.

As you see, I raise the questions about using the pqa command which at the moment is barely usable with ollama.

I am sorry I can't provide outputs and command details right now, but I promise I will post the details because I like paper-qa.

@Snikch63200 Please continue exploring ollama and paper-qa, I learn a lot from your questions and posts. I appreciate it. I will also update with the statistics of my usage.

[maykcaldas]

@rosegarden-coder

Yeah, using ollama/nomic-embed-text for embeddings works fine.

Now, let me answer these two raised points:

1. I meant that if one wants to use anthropic while running the pqa command line interface (cli), one must configure it too. Notice that only setting the ANTHROPIC_API_KEY environment variable will not flag to PaperQA to use anthropic models. PaperQA will still use the default models, which are OpenAI's.
   A setting is also needed to use anthropic models.

For instance, this command runs ollama in the cli, which I imagine is what you referred to as "setting Kungfu":

pqa --llm "ollama/llama3.2" \
    --summary_llm "ollama/llama3.2" \
    --agent.agent_llm "ollama/llama3.2" \
    --embedding "ollama/mxbai-embed-large" \
    --llm_config '{"model_list": [{"model_name": "ollama/llama3.2", "litellm_params": {"model": "ollama/llama3.2", "api_base": "http://localhost:11434"}}]}' \
    --summary_llm_config '{"model_list": [{"model_name": "ollama/llama3.2", "litellm_params": {"model": "ollama/llama3.2", "api_base": "http://localhost:11434"}}]}' \
    --agent.agent_llm_config '{"model_list": [{"model_name": "ollama/llama3.2", "litellm_params": {"model": "ollama/llama3.2", "api_base": "http://localhost:11434"}}]}' \
    --prompts.use_json False ask "Your question here"

And I agree. Invoking this huge command line whenever you want to run pqa is unacceptable. However, what I meant by suggesting using pqa -s <your_setting> was to save this setting:

pqa --llm "ollama/llama3.2" \
    --summary_llm "ollama/llama3.2" \
    --agent.agent_llm "ollama/llama3.2" \
    --embedding "ollama/mxbai-embed-large" \
    --llm_config '{"model_list": [{"model_name": "ollama/llama3.2", "litellm_params": {"model": "ollama/llama3.2", "api_base": "http://localhost:11434"}}]}' \
    --summary_llm_config '{"model_list": [{"model_name": "ollama/llama3.2", "litellm_params": {"model": "ollama/llama3.2", "api_base": "http://localhost:11434"}}]}' \
    --agent.agent_llm_config '{"model_list": [{"model_name": "ollama/llama3.2", "litellm_params": {"model": "ollama/llama3.2", "api_base": "http://localhost:11434"}}]}' \
      save ollama

This command will save a ollama.json setting in your PaperQA home. Then, the usage becomes as simple as:

pqa -s ollama ask "Your question here"

Therefore, upon saving the setting, one can run the pqa cli with ollama.
I could create an "ollama_default" config, but it might not be interesting because there are many llama flavors (llmama3.1, Llama3.3+granite3-dense:8b, ...). It's not worth forcing a default ollama model without first validating it.

2. You are right. One cannot run ollama without editing Contracrow's setting.

In pqa, -s' is short for --settings. It invokes a pre-defined setting from a [pool of available settings](https://github.com/Future-House/paper-qa?tab=readme-ov-file#bundled-settings). Not outstanding, -s contracrowuses a validated setting for this task. You can see the actual config [here](https://github.com/Future-House/paper-qa/blob/main/paperqa/configs/contracrow.json). But notice that you *can* change this setting to use ollama models, save it to your PaperQA home settings (where the option -s' checks the settings and use your adapted contracrow setting.
In PaperQA, a setting configures the agent. The LLMs used are vital to the agent, so they are part of the settings and can be changed by the user.

I can run -s contracrow normally. But, because this default setting uses claude-3-5-sonnet-20240620, I can use it if I define an ANTHROPIC_API_KEY environment variable. But the env variable solely does not make PaperQA use anthropic models.

Forgive my extensive answer, but I wanted to embase this before saying that I disagree that pqa's cli is barely usable with ollama. This is just a matter of configuring your setting setup in the same way it is needed for any non-default provider.

Thank you very much for fomenting this discussion, pointing out where our documentation is unclear, and showing which new features PaperQA needs. It is much appreciated!

Please let me know if this answer works for you or if you have a suggestion on how this interface should work. But because the setting is the model configuration, I don't think it should be API agnostic. The setting allows the user to decide which provider/model to use.

[rosegarden-coder]

@maykcaldas

Please be as extensive as possible so that all users can learn. Allow me to reply with three points:

First, the (ideal) expected usage of pqa for ollama could be (in an imaginary documentation):

PaperQA2 uses an LLM to operate, so you'll need to either set an appropriate API key environment variable (i.e. export OPENAI_API_KEY=sk-...) or set up an open source LLM server (export OLLAMA=127.0.0.1:11434, llama3.2, mxbai-embed-large)

In this way, one can start using pqa with ollama as equally and as rapidly as they do with those corporate APIs. Unfortunately, users can not do it, and then they go to the backup plan.

Second, I tried the exact settings you posted above, but unfortunately, it does not work. It can not index and read the paper. This is what I did:

pqa --llm "ollama/llama3.2" \
    --summary_llm "ollama/llama3.2" \
    --agent.agent_llm "ollama/llama3.2" \
    --embedding "ollama/mxbai-embed-large" \
    --llm_config '{"model_list": [{"model_name": "ollama/llama3.2", "litellm_params": {"model": "ollama/llama3.2", "api_base": "http://localhost:11434"}}]}' \ 
    --summary_llm_config '{"model_list": [{"model_name": "ollama/llama3.2", "litellm_params": {"model": "ollama/llama3.2", "api_base": "http://localhost:11434"}}]}' \
    --agent.agent_llm_config '{"model_list": [{"model_name": "ollama/llama3.2", "litellm_params": {"model": "ollama/llama3.2", "api_base": "http://localhost:11434"}}]}' \
      save ollama
[19:36:26] Settings saved to: /home/x/.pqa/settings/ollama.json

then I enter the paper dir cd pe, and did this:

pqa -s ollama ask "main idea of the paper"
Could not find cost for model ollama/llama3.2.
Encountered exception during tool call for tool gather_evidence: EmptyDocsError('Not gathering evidence due to having no papers.')
Could not find cost for model ollama/llama3.2.
[19:47:09] Completing 'main idea of the paper' as 'certain'.                                                                                                  
           Generating answer for 'main idea of the paper'.                                                                                                    
Could not find cost for model ollama/llama3.2.
           Status: Paper Count=0 | Relevant Papers=0 | Current Evidence=0 | Current Cost=$0.0000                                                              
[19:47:10] Answer: I cannot provide an answer without the context. Please provide the context for the question "main idea of the paper", and I'll be happy to 
           assist you in writing a concise and accurate response in the style of a Wikipedia article.                                                         
/home/x/Micromamba/envs/paperQA517/lib/python3.12/site-packages/pydantic/main.py:426: UserWarning: Pydantic serializer warnings:
  Expected `bool` but got `str` with value `'False'` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(

As you see. It does not index the paper so it can not answer. I wonder what is the output on your side when you run the code above.

Then I tried to tackle the problem by adding the paper path manually and create a new setting by adding one extra line to designate the dir path:

pqa --llm "ollama/llama3.2" \
    --summary_llm "ollama/llama3.2" \
    --agent.agent_llm "ollama/llama3.2" \
    --embedding "ollama/mxbai-embed-large" \
    --llm_config '{"model_list": [{"model_name": "ollama/llama3.2", "litellm_params": {"model": "ollama/llama3.2", "api_base": "http://localhost:11434"}}]}' \
    --summary_llm_config '{"model_list": [{"model_name": "ollama/llama3.2", "litellm_params": {"model": "ollama/llama3.2", "api_base": "http://localhost:11434"}}]}' \
    --agent.agent_llm_config '{"model_list": [{"model_name": "ollama/llama3.2", "litellm_params": {"model": "ollama/llama3.2", "api_base": "http://localhost:11434"}}]}' \
    **--agent.index.paper_directory "/home/x/pe" \**
    save ollama2

This time, it did see the paper, but it also failed to continue:

Your model may not be capable of supporting JSON output or our parsing technique could use some work. Try a different model or specify Settings(prompts={\'use_json\': False})')
Could not find cost for model ollama/llama3.2.
Encountered exception during tool call for tool gen_answer: TypeError("GenerateAnswer.gen_answer() got an unexpected keyword argument 'question'")

so... llama3.2 does not work. I am going to use llama3.3 anyway, so again I changed the settings to this:

pqa --llm "ollama/llama3.3" \
    --summary_llm "ollama/llama3.3" \
    --agent.agent_llm "ollama/llama3.3" \
    --embedding "ollama/mxbai-embed-large" \
    --llm_config '{"model_list": [{"model_name": "ollama/llama3.3", "litellm_params": {"model": "ollama/llama3.3", "api_base": "http://localhost:11434"}}]}' \ 
    --summary_llm_config '{"model_list": [{"model_name": "ollama/llama3.3", "litellm_params": {"model": "ollama/llama3.3", "api_base": "http://localhost:11434"}}]}' \
    --agent.agent_llm_config '{"model_list": [{"model_name": "ollama/llama3.3", "litellm_params": {"model": "ollama/llama3.3", "api_base": "http://localhost:11434"}}]}' \
    --agent.index.paper_directory "/home/x/pe" \
    save ollama3

and then pqa -s ollama3 ask "main idea of the paper". Finally, it worked not llama3.2, but llama3.3 and paper-qa version 5.17 when I tested with one paper. If there are multiple papers (<10), I encounter messages like this in ollama output.

time=2025-03-07T20:33:13.798+08:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.218471258 model=/home/x/.ollama/models/blobs/ -- sha256-819c2adf5ce6df2b6bd2ae4ca90d2a69f060afeb438d0c171db57daa02e39c3d --- not sure why this happens
time=2025-03-07T20:29:45.827+08:00 level=WARN source=runner.go:129 msg="truncating input prompt" limit=2048 prompt=2152 keep=5 new=2048 --- this maybe related to the model's context size, I am testing further by increasing the model context size.

To sum: it works with llama3.3 and setting the paper dir and with one paper (not multiple papers, see below)

Third, now, further configuring Bundled Settings such as contracrow.json is too daunting for me now. But these bundled settings are very useful. Could you test the bundle settings, particularly contracrow.json to see how to make it work with ollama?

I appreciate the project and the work of the team.

[rosegarden-coder]

Further testing with only 8 papers, there is a problem of timing out and it took an extremely long time and fail:

time=2025-03-07T21:00:01.602+08:00 level=INFO source=server.go:725 msg="aborting completion request due to client closing the connection" from ollama

Encountered exception during tool call for tool gather_evidence: litellm.APIConnectionError: OllamaException - litellm.Timeout: Connection timed out. Timeout passed=60.0, time taken=60.039 seconds. Received Model Group=ollama/llama32768
Available Model Group Fallbacks=None LiteLLM Retried: 2 times, LiteLLM Max Retries: 3 -- from paper-pa

Agent timeout after 500.0-sec, just answering. from paper-qa

I am attaching the setting json file for your thorough review:
ollama3.json

and here is the failure message:

^[[CEncountered exception during tool call for tool gen_answer: litellm.APIConnectionError: OllamaException - litellm.Timeout: Connection timed out. Timeout passed=60.0, time taken=60.012 seconds. Received Model Group=ollama/llama32768
Available Model Group Fallbacks=None LiteLLM Retried: 2 times, LiteLLM Max Retries: 3

[Snikch63200]

Hi @rosegarden-coder ,

Half-sucessfully tested Llama3.3+granite3-dense:8b with both models are served on ollama server.

The first problem you noticed :

time=2025-03-07T20:33:13.798+08:00 level=WARN source=sched.go:646 msg="gpu VRAM usage didn't recover within timeout" seconds=5.218471258 model=/home/x/.ollama/models/blobs/ -- sha256-819c2adf5ce6df2b6bd2ae4ca90d2a69f060afeb438d0c171db57daa02e39c3d --- not sure why this happens
time=2025-03-07T20:29:45.827+08:00 level=WARN source=runner.go:129 msg="truncating input prompt" limit=2048 prompt=2152 keep=5 new=2048 --- this maybe related to the model's context size, I am testing further by increasing the model context size.

It's typically a parameter problem (that's why, i created this discussion about parameters considering informations about LLM 'ideal' parameters are very hard to find). I guess by increasing context size.

For Llama3.3+granite3-dense:8b, I use these :

local_llm_config = dict(
                        model_list=[
                                    dict(
                                        model_name="ollama/llama3.3:latest",
                                        litellm_params=dict(
                                                            model="ollama/llama3.3:latest",
                                                            api_type= "ollama",
                                                            api_base="http://localhost:11434",
                                                            stream=True,
                                                            caching=True, #does nothing...
                                                            cache= True,
                                                            mirostat=0,
                                                            mirostat_eta=0.1,
                                                            mirostat_tau=5.0,
                                                            num_ctx=32000, #Default 2048
                                                            repeat_last_n=64, # Sets how far back for the model to look back to prevent repetition. (Default: 64, 0 = disabled, -1 = num_ctx)
                                                            repeat_penalty=1.1,
                                                            temperature=0.5,
                                                            seed=0,
                                                            tfs_z=1,
                                                            num_predict=32000,# Maximum number of tokens to predict when generating text. (Default: 128, -1 = infinite generation, -2 = fill context)
                                                            top_k=40,
                                                            top_p=0.9,
                                                            min_p=0.0,
                                                            timeout = 600,
                                                            batch_size = 1024,
                                                            num_keep = 512,
                                                            ),
                                        ),

                                    ]
                        )

Notice :

• batch_size = 1024 seems increase answer relevance (don't know why... if someone can explain...)
• num_keep = 512 also seems increase answer relevance (don't know why... if someone can explain...)
• num_ctx=32000 when you increase sources number you increase tokens injected in request so num_ctx have to be increased too. Llama 3.3 have a 128k context limit (read this in papers)
• num_predict=32000. If your output token limit is not sufficent, output JSON will be truncated and leads to an error (spent a lot of time to understant this point ;-)). Often, 4096 or 8192 are enough.

Second problem you noticed :

Encountered exception during tool call for tool gather_evidence: litellm.APIConnectionError: OllamaException - litellm.Timeout: Connection timed out. Timeout passed=60.0, time taken=60.039 seconds. Received Model Group=ollama/llama32768
Available Model Group Fallbacks=None LiteLLM Retried: 2 times, LiteLLM Max Retries: 3 -- from paper-pa

-> Just increase 'timeout' to 600 in parameters (see above).

For me, combination of Llama3.3+granite3-dense:8b leads to agent timeout (set to 3200 s.) but gives relevant answer for a few papers (evidence_k = 5, answer_max_sources = 3). I guess it is a hardware limitation...

In bief, Llama3.3+granite3-dense:8b returns much much accurate results than 'dolphin-mixtral:8x7b+mxbai-embed-large' but is much much ressource greedy too...

Best regards.

[rosegarden-coder]

@Snikch63200 I appreciate your guidance. I have been very busy during the week and I will test your recommendations later.

Can you look at the attached json file? ollama3.json

Can I put the settings you suggested into this file and make it work -- running pqa -s command?

[maykcaldas]

Hello @rosegarden-coder and @Snikch63200 . Thanks for looking into this further.
I have been busy these last couple weeks but plan to look into this soon.

Let me quickly answer a few of your questions and I will get back to the other ones later. Is that ok?

First, the (ideal) expected usage of pqa for ollama could be (in an imaginary documentation):

PaperQA2 uses an LLM to operate, so you'll need to either set an appropriate API key environment variable (i.e. export OPENAI_API_KEY=sk-...) or set up an open source LLM server (export >OLLAMA=127.0.0.1:11434, llama3.2, mxbai-embed-large)

Notice that using this kind of interface, we would allow configuration through the environment variable. PaperQA does not do that. All the configuration of the agents/llms is done through the Settings class. Therefore, I think this is a mischaracterization of current PaperQA behavior. Setting OPENAI_API_KEY is not a configuration, it's just a required key to use the default models. The used can still use Settings to select a specific OpenAI model. I understand that it may be easier to configure through a environment variable, but given the number of customizable parameters, I think it'll quickly become unpractical.

Second, I tried the exact settings you posted above, but unfortunately, it does not work. It can not index and read the paper. This is what I did:
(...)

This is an actual bug. What happens is: when you save the settings, PaperQA is saving a whole Settings object, which include defaults for the agent.index.paper_directory and agent.index.index_directory (default is the env variable PQA_HOME or user home case PQA_HOME is not set). Your solution of specifying the paper_directory is a good workaround it. But should definitely be improved. Thanks for catching this.

Your model may not be capable of supporting JSON output or our parsing technique could use some work. Try a different model or specify Settings(prompts={'use_json': False})')
Could not find cost for model ollama/llama3.2.
Encountered exception during tool call for tool gen_answer: TypeError("GenerateAnswer.gen_answer() got an unexpected keyword argument 'question'")

As greatly noticed by @Snikch63200 this is a misbehavior of the LLM itself. It seems that Llama failed to complete a valid json of follow the correct schema. I could reproduce this error when I tried to run the exact same command multiple times. Llama3.2 indeed failed sometimes. This is not a PaperQA error.

Finally, I'll some more time to look into details of this timeout error as I couldn't reproduce.

I will also get back to you on the contracrow settings with ollama. Just a quick answer on that: you should be able to use a "base setting" to save your setting. Meaning that you can do something like pqa -s contrarow --llm ollama/llama3.2 (...) save ollama_contracrow and it'll start from contracrow and change the hyperparameters you passed. I didn't test it specifically for ollama and contracrow, but we tested this feature in other cases.

Thanks for your patience on that matter , I'll get back to you asap. In the meantime, thank you so much. @Snikch63200 for the support here

[rosegarden-coder]

Hello @Snikch63200 For the spirit of libre software, can I shamelessly ask you to share your script and configuration? I see that you have dug deeper into this topic.