All queries return "Relevant Papers=0"

[BenWibking]

I am using paper-qa locally with llama3.3 with these settings:

pqa --llm "ollama/llama3.3" \
    --summary_llm "ollama/llama3.3" \
    --agent.agent_llm "ollama/llama3.3" \
    --embedding "ollama/granite3-dense:8b" \
    --llm_config '{"model_list": [{"model_name": "ollama/llama3.3", "litellm_params": {"model": "ollama/llama3.3", "api_base": "http://localhost:11434"}}]}' \
    --summary_llm_config '{"model_list": [{"model_name": "ollama/llama3.3", "litellm_params": {"model": "ollama/llama3.3", "api_base": "http://localhost:11434"}}]}' \
    --agent.agent_llm_config '{"model_list": [{"model_name": "ollama/llama3.3", "litellm_params": {"model": "ollama/llama3.3", "api_base": "http://localhost:11434"}}]}' \
      save ollama

I have downloaded two review articles in my field, and ask very generic questions with keywords that appear in article titles.

I run a search for "circumgalactic medium" and it finds the two articles I've indexed:

$ pqa -i science search "circumgalactic medium"
                                          PDF Search
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Title                                               ┃ File                                 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Key Physical Processes in the Circumgalactic Medium │ Faucher-Giguere_2023_ARAA_61_131.pdf │
│ The Cosmic Baryon and Metal Cycles                  │ Peroux_2020_ARAA_58_363.pdf          │
└─────────────────────────────────────────────────────┴──────────────────────────────────────┘

But asking it "what is the circumgalactic medium?" results in an infinite loop of nothing, saying "Relevant Papers=0":

$ pqa -s ollama -i science ask 'What is the circumgalactic medium?'
Could not find cost for model ollama/llama3.3.
[12:05:58] Starting paper search for 'circumgalactic medium'.
           paper_search for query 'circumgalactic medium' and offset 0 returned 2 papers.
           Status: Paper Count=2 | Relevant Papers=0 | Current Evidence=0 | Current Cost=$0.0000
Could not find cost for model ollama/llama3.3.
[12:05:59] Starting paper search for 'circumgalactic medium'.
           paper_search for query 'circumgalactic medium' and offset 8 returned 0 papers.
           Status: Paper Count=2 | Relevant Papers=0 | Current Evidence=0 | Current Cost=$0.0000
Could not find cost for model ollama/llama3.3.
[12:06:00] Starting paper search for 'circumgalactic medium'.
           paper_search for query 'circumgalactic medium' and offset 0 returned 2 papers.
           Status: Paper Count=2 | Relevant Papers=0 | Current Evidence=0 | Current Cost=$0.0000
Could not find cost for model ollama/llama3.3.
[12:06:01] Starting paper search for 'circumgalactic medium'.
           paper_search for query 'circumgalactic medium' and offset 16 returned 0 papers.
           Status: Paper Count=2 | Relevant Papers=0 | Current Evidence=0 | Current Cost=$0.0000
Could not find cost for model ollama/llama3.3.
[12:06:02] Starting paper search for 'circumgalactic medium'.
           paper_search for query 'circumgalactic medium' and offset 24 returned 0 papers.
           Status: Paper Count=2 | Relevant Papers=0 | Current Evidence=0 | Current Cost=$0.0000

What is going on here?

[jamesbraza]

So to get "Relevant Papers" above 0 the agent needs to call the gather_evidence tool. In the trace you've included, the agent never called that tool. Are you able to run the agent for more steps?

The "Could not find cost for model ollama/llama3.3" is related to BerriAI/litellm#9644

[BenWibking]

Where are the logs written? The CLI stdout output does not show this.

[jamesbraza]

In your printouts above:

[15:41:48] Starting paper search for 'CGM observations simulations feedback mechanisms tension'.
           paper_search for query 'CGM observations simulations feedback mechanisms tension' and offset 0 returned 2 papers.
           Status: Paper Count=2 | Relevant Papers=2 | Current Evidence=12 | Current Cost=$0.1151
[15:41:55] Starting paper search for 'CGM observations simulations feedback mechanisms tension'.
           paper_search for query 'CGM observations simulations feedback mechanisms tension' and offset 8 returned 0 papers.
           Status: Paper Count=2 | Relevant Papers=2 | Current Evidence=12 | Current Cost=$0.1283

You can see the paper_search tool was called with its arguments, and its returned status

[BenWibking]

Ok, the logs written to standard output do not show complete being called.

[BenWibking]

When I use Claude 3.7 for all LLMs (except the embedding model), I get a similar timeout, but no answer at all.

These are my settings:

pqa --llm "claude-3-7-sonnet-20250219" \
    --summary_llm "claude-3-7-sonnet-20250219" \
    --summary_llm_config '{"rate_limit": {"claude-3-7-sonnet-20250219": "20000 per 1 minute"}}' \
    --agent.agent_llm "claude-3-7-sonnet-20250219" \
    --agent.timeout 300.0 \
    --embedding "ollama/granite3-dense:8b" \
      save claude

This is the full CLI output:

review_articles git:(main) ✗ uv run pqa -s claude ask "What are the tensions between CGM observations and simulations?"
[17:38:16] Starting paper search for 'tensions between CGM observations and simulations'.
           paper_search for query 'tensions between CGM observations and simulations' and offset 0 returned 2 papers.
           Status: Paper Count=2 | Relevant Papers=0 | Current Evidence=0 | Current Cost=$0.0049
[17:38:19] gather_evidence starting for question 'What are the tensions between CGM observations and simulations?'.
[17:40:27] Status: Paper Count=2 | Relevant Papers=0 | Current Evidence=0 | Current Cost=$0.1676
[17:40:29] Starting paper search for 'circumgalactic medium observations simulations discrepancies'.
           paper_search for query 'circumgalactic medium observations simulations discrepancies' and offset 0 returned 2 papers.
           Status: Paper Count=2 | Relevant Papers=0 | Current Evidence=0 | Current Cost=$0.1736
[17:40:30] Starting paper search for 'circumgalactic medium observations simulations discrepancies'.
           paper_search for query 'circumgalactic medium observations simulations discrepancies' and offset 8 returned 0 papers.
           Status: Paper Count=2 | Relevant Papers=0 | Current Evidence=0 | Current Cost=$0.1801
[17:40:32] Starting paper search for '"CGM observations vs simulations" discrepancies challenges'.
           paper_search for query '"CGM observations vs simulations" discrepancies challenges' and offset 0 returned 2 papers.
           Status: Paper Count=2 | Relevant Papers=0 | Current Evidence=0 | Current Cost=$0.1870
[17:40:36] Resetting 'What are the tensions between CGM observations and simulations?'.
[17:40:39] Starting paper search for 'circumgalactic medium CGM observations simulations comparison'.
           paper_search for query 'circumgalactic medium CGM observations simulations comparison' and offset 0 returned 2 papers.
           Status: Paper Count=2 | Relevant Papers=0 | Current Evidence=0 | Current Cost=$0.2008
[17:40:41] gather_evidence starting for question 'What are the tensions between CGM observations and simulations?'.
Encountered exception during tool call for tool gather_evidence: TimeoutError("Timeout (60.00999999999663 secs): rate limit for key: ('client', 'claude-3-7-sonnet-20250219')")
[17:42:02] gather_evidence starting for question 'What are the tensions between CGM observations and simulations?'.
[17:43:15] Agent timeout after 300.0-sec, just answering.
/Users/benwibking/.local/share/uv/python/cpython-3.11.9-macos-aarch64-none/lib/python3.11/asyncio/base_events.py:1937: RuntimeWarning: coroutine 'map_fxn_summary' was never awaited
  handle = None  # Needed to break cycles when an exception occurs.
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
           Generating answer for 'What are the tensions between CGM observations and simulations?'.
Encountered exception during tool call for tool gen_answer: TimeoutError("Timeout (60.00999999999663 secs): rate limit for key: ('client', 'claude-3-7-sonnet-20250219')")
[17:44:30] Answer:

[jamesbraza]

We unit test the complete tool to be callable and work, so you need to root cause why the agent LLM isn't ever selecting the complete tool.

It may be that the 12 ranked contextual summaries (per the Current Evidence=12 in your logs) made by the gather_evidence tool, the agent is thinking they aren't sufficient, so it keeps running paper_search to get new ones. You'll have to root cause further yourself given your own data