[BUG] respect_context_window doesn't work

[maxkochanoff]

Description

Sometimes I run into a litellm error about exceeding the context window, even though the respect_context_window parameter is set to True by default. I thought respect_context_window was supposed to prevent that by automatically summarizing the text. So now I'm not sure if it's a bug or if I'm just doing something wrong. I would be very grateful if you could help me please.

P.S. In my setup, I do have some tools that are pretty lengthy (but definitely still under the maximum context length). Maybe the problem occurs when an Agent retries to call a tool after a failure (in this case there would be a lot of text in the messages).

Steps to Reproduce

None

Expected behavior

No errors

Screenshots/Code snippets

None

Operating System

Ubuntu 20.04

Python Version

3.10

crewAI Version

0.114.0

crewAI Tools Version

0.40.1

Virtual Environment

Venv

Evidence

The exact error:

litellm.ContextWindowExceededError: litellm.BadRequestError: ContextWindowExceededError: MistralException - Error code: 400 - {'object': 'error', 'message': "This model's maximum context lenght is 15000 tokens. However, you requested 15018 tokens (12970) in the messages, 2048 in the completion). Please reduce the length of the messages or completion.", type: 'BadRequestError', 'param': None, 'code': 400}

Possible Solution

None

Additional context

None

[Vidit-Ostwal]

Ideally, it should get summarise, can you share some logs, with setting verbose=True.

[lucasgomide]

@maxkochanoff are you using any memory?

[maxkochanoff]

@lucasgomide I don't think so. I haven't used any specific parameters for that. However, it might be possible if some of these parameters have default values.

@Vidit-Ostwal There aren't a lot of logs, to be honest. I think that the crash happens when the agent needs to do the function calling.

First, the logs start with messages that include my lengthy tools` descriptions, which I implemented by inheriting from the BaseTool class.

Then I have this traceback:

Traceback (most recent call last):
    File "venv/lib/python3.10/site-packages/openai/_base_client.py", line 1036, in _request
        response.raise_for_status()
    File "venv/lib/python3.10/site-packages/httpx/_models.py", line 763, in raise_for_status
        raise HTTPStatusError(message, request=request, response=self)
litellm.ContextWindowExceededError: litellm.BadRequestError: ContextWindowExceededError: MistralException - Error code: 400 - {'object': 'error', 'message': "This model's maximum context lenght is 15000 tokens. However, you requested 15018 tokens (12970) in the messages, 2048 in the completion). Please reduce the length of the messages or completion.", type: 'BadRequestError', 'param': None, 'code': 400}

Also I have several errors like that (but it isn't the cause of the crash):

utils.py:4355 - Error getting model info: This model isn't mapped yet.

I'm not sure, but it seems to me that when this second error happens, CrewAI retries the request to the model. Given that my tools' descriptions are lengthy, the context grows very quickly (because each retry repeats the prompt along with the tools' descriptions). In the end, it crashes due to the inability to summarise (or truncate) the context.

[lucasgomide]

any chance to share a snippet code here?

[maxkochanoff]

@lucasgomide My crew is fairly standard, based on the basic examples in documentation.

My tools follow this example. Nothing special. However, the descriptions of my tools are very detailed, with fairly lengthy documentation, which definitely increases the token count. My inputs are also lengthy (can be 400+ words).

Here a small code snippet:

llm = LLM(model=..., base_url=..., supports_system_messages=False, max_tokens=2048, api_key='None')

@CrewBase
class MyCrew():
    @agent
    def agent_one(self):
        return Agent(
            role=...,
            goal=...,
            backstory=...,
            use_system_prompt=False,
            llm=llm
        )

    @task
    def task_one(self):
        return Task(
            description=...,
            expected_output=...,
            agent=agent_one,
            output_pydantic=MyPydanticModelOne,
    )

    @task 
    def task_two(self):
        return ConditionalTask(
            condition=...,
            description=...,
            expected_output=...,
            agent=agent_one,
            tools=my_tools,
            output_pydantic=MyPydanticModelTwo
        )

def run(my_inputs):
  """
  Run the crew.
  """
  inputs = {
    'inputs': my_inputs
  }
  MyCrew().crew().kickoff(inputs=inputs)

[lucasgomide]

I’ll take a look at that by next week.
Anyway, is there any chance you could summarize the information (input and tool description) in advanced?

[Vidit-Ostwal]

@lucasgomide My crew is fairly standard, based on the basic examples in documentation.

My tools follow this example. Nothing special. However, the descriptions of my tools are very detailed, with fairly lengthy documentation, which definitely increases the token count. My inputs are also lengthy (can be 400+ words).

Here a small code snippet:

llm = LLM(model=..., base_url=..., supports_system_messages=False, max_tokens=2048, api_key='None')

@CrewBase
class MyCrew():
    @agent
    def agent_one(self):
        return Agent(
            role=...,
            goal=...,
            backstory=...,
            use_system_prompt=False,
            llm=llm
        )

    @task
    def task_one(self):
        return Task(
            description=...,
            expected_output=...,
            agent=agent_one,
            output_pydantic=MyPydanticModelOne,
    )

    @task 
    def task_two(self):
        return ConditionalTask(
            condition=...,
            description=...,
            expected_output=...,
            agent=agent_one,
            tools=my_tools,
            output_pydantic=MyPydanticModelTwo
        )

def run(my_inputs):
  """
  Run the crew.
  """
  inputs = {
    'inputs': my_inputs
  }
  MyCrew().crew().kickoff(inputs=inputs)

@maxkochanoff
do you see any such logs in the when you set respect_context_window parameter as True.

Context length exceeded. Summarizing content to fit the model context window.

[Vidit-Ostwal]

@lucasgomide, I can work on this one, if you don't mind can you assign this to me?

[maxkochanoff]

@Vidit-Ostwal no, I shared all logs I had

[Vidit-Ostwal]

@maxkochanoff,
Can you try this once,

set this parameter called litellm._turn_on_debug()
and share all the logs with us, along with setting verbose = True at agent and crew level.

This will help us to understand what requests are getting made to litellm.
thanks