Skip to content

[BUG]: Intermittent failures with LLM agents example #1422

Open

Description

Version

24.03

Which installation method(s) does this occur on?

Source

Describe the bug.

Hitting an intermittent error:

 Exception msg: ValueError: LLMMathChain._evaluate("
age**(0.43)
") raised error: 'age'. Please try again with a valid numerical expression

Minimum reproducible example

python examples/llm/main.py --log_level=debug agents simple


### Relevant log output

<details><summary>Click here to see error details</summary><pre>

> Entering new AgentExecutor chain...
 I should use a search engine to find information about Leo DiCaprio's girlfriend.
Action: Search
Action Input: "Leo DiCaprio girlfriend"
Observation: Leonardo DiCaprio and Girlfriend Vittoria Ceretti Are 'In Love': Source | UsWeekly. By clicking Sign Up, you agree to our Terms and Conditions and that you have read our Privacy Policy.
Source rate[Complete]: 1 questions [00:00, 110.96 questions/s] I should use a calculator to calculate the girlfriend's current age raised to the 0.43 power.
Action: Calculator
Source rate[Complete]: 1 questions [00:00, 110.96 questions/s]E20231211 15:41:10.464584 187172 runnable.hpp:112] /linear_segment_0/llm-engine-3; rank: 0; size: 1; tid: 140383245805248 Unhandled exception occurred. Rethrowing
E20231211 15:41:10.464756 187172 context.cpp:124] /linear_segment_0/llm-engine-3; rank: 0; size: 1; tid: 140383245805248: set_exception issued; issuing kill to current runnable. Exception msg: ValueError: LLMMathChain._evaluate("
age**(0.43)
") raised error: 'age'. Please try again with a valid numerical expression

At:
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(87): _evaluate_expression
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(125): _aprocess_llm_result

E20231211 15:41:10.465035 187043 runner.cpp:189] RuE20231211 15:41:10.465302 187043 segment_instance.cpp:273] segment::SegmentInstance - an exception was caught while awaiting on one or more nodes - rethrowing[00:00, ? events/s]
E20231211 15:41:10.465325 187043 service.cpp:224] Service[segment::SegmentInstance]: caught exception in service_await_join: ValueError: LLMMathChain._evaluate("
age**(0.43)
") raised error: 'age'. Please try again with a valid numerical expression

At:
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(87): _evaluate_expression
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(125): _aprocess_llm_result
E20231211 15:41:10.465353 187043 pipeline_instance.cpp:230] pipeline::PipelineInstance - an exception was caught while awaiting on segments - rethrowing
E20231211 15:41:10.465374 187043 service.cpp:224] Service[pipeline::PipelineInstance]: caught exception in service_await_join: ValueError: LLMMathChain._evaluate("
age**(0.43)
") raised error: 'age'. Please try again with a valid numerical expression

At:
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(87): _evaluate_expression
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(125): _aprocess_llm_result
E20231211 15:41:10.465499 187043 service.cpp:224] Service[pipeline::Manager]: caught exception in service_await_join: ValueError: LLMMathChain._evaluate("
age**(0.43)
") raised error: 'age'. Please try again with a valid numerical expression

At:
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(87): _evaluate_expression
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(125): _aprocess_llm_result
E20231211 15:41:10.465521 187043 service.cpp:224] Service[ExecutorDefinition]: caught exception in service_await_join: ValueError: LLMMathChain._evaluate("
age**(0.43)
") raised error: 'age'. Please try again with a valid numerical expression

At:
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(87): _evaluate_expression
  /home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py(125): _aprocess_llm_result
Source rate[Complete]: 1 questions [00:00, 110.96 questions/s]
Upload rate[Complete]: 0 events [00:00, ? events/s]
Exception occurred in pipeline. Rethrowing
Traceback (most recent call last):
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/numexpr/necompiler.py", line 760, in getArguments
    a = local_dict[name]
KeyError: 'age'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py", line 80, in _evaluate_expression
    numexpr.evaluate(
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/numexpr/necompiler.py", line 975, in evaluate
    raise e
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/numexpr/necompiler.py", line 874, in validate
    arguments = getArguments(names, local_dict, global_dict, _frame_depth=_frame_depth)
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/numexpr/necompiler.py", line 762, in getArguments
    a = global_dict[name]
KeyError: 'age'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dagardner/work/morpheus/morpheus/pipeline/pipeline.py", line 419, in join
    await executor.join_async()
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/asyncio/tasks.py", line 650, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "/home/dagardner/work/morpheus/morpheus/llm/nodes/langchain_agent_node.py", line 75, in execute
    results = await self._run_single(**input_dict)
  File "/home/dagardner/work/morpheus/morpheus/llm/nodes/langchain_agent_node.py", line 61, in _run_single
    results = await asyncio.gather(*results_async)
  File "/home/dagardner/work/morpheus/morpheus/llm/nodes/langchain_agent_node.py", line 69, in _run_single
    return await self._agent_executor.arun(**kwargs)
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/base.py", line 266, in arun
    return (await self.acall(kwargs, callbacks=callbacks))[self.output_keys[0]]
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/base.py", line 178, in acall
    raise e
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/base.py", line 172, in acall
    await self._acall(inputs, run_manager=run_manager)
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/agents/agent.py", line 1002, in _acall
    next_step_output = await self._atake_next_step(
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/agents/agent.py", line 928, in _atake_next_step
    result = await asyncio.gather(
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/agents/agent.py", line 909, in _aperform_agent_action
    observation = await tool.arun(
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/tools/base.py", line 358, in arun
    raise e
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/tools/base.py", line 330, in arun
    await self._arun(*tool_args, run_manager=run_manager, **tool_kwargs)
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/tools/base.py", line 430, in _arun
    await self.coroutine(
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/base.py", line 263, in arun
    return (await self.acall(args[0], callbacks=callbacks))[self.output_keys[0]]
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/base.py", line 178, in acall
    raise e
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/base.py", line 172, in acall
    await self._acall(inputs, run_manager=run_manager)
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py", line 163, in _acall
    return await self._aprocess_llm_result(llm_output, _run_manager)
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py", line 125, in _aprocess_llm_result
    output = self._evaluate_expression(expression)
  File "/home/dagardner/work/conda/envs/morpheus/lib/python3.10/site-packages/langchain/chains/llm_math/base.py", line 87, in _evaluate_expression
    raise ValueError(
ValueError: LLMMathChain._evaluate("
age**(0.43)
") raised error: 'age'. Please try again with a valid numerical expression


</pre></details>


### Full env printout

<details><summary>Click here to see environment details</summary><pre>

 [Paste the results of print_env.sh here, it will be hidden by default]

</pre></details>


### Other/Misc.

_No response_

### Code of Conduct

- [X] I agree to follow Morpheus' Code of Conduct
- [X] I have searched the [open bugs](https://github.com/nv-morpheus/Morpheus/issues?q=is%3Aopen+is%3Aissue+label%3Abug) and have found no duplicates for this bug report
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingsherlockIssues/PRs related to Sherlock workflows and components

    Type

    No type

    Projects

    • Status

      Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions