[Bug]: IndexError in utils.py

[paul-gauthier]

What happened?

A user reported the following traceback of an IndexError. This got promoted to a litellm.exceptions.APIConnectionError.

How should I treat litellm.exceptions.APIConnectionErrors? Should I retry them or consider them to be fatal?

Here is the original issue filed by an aider user:
Aider-AI/aider#576

Relevant log output

Traceback (most recent call last):
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/litellm/utils.py", line 9735, in chunk_creator
    if hasattr(chunk.parts[0], "finish_reason"):
               ~~~~~~~~~~~^^^
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/proto/marshal/collections/repeated.py", line 125, in __getitem__
    return self._marshal.to_python(self._pb_type, self.pb[key])
                                                  ~~~~~~~^^^^^
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/litellm/utils.py", line 9740, in chunk_creator
    if chunk.parts[0].finish_reason.name == "SAFETY":
       ~~~~~~~~~~~^^^
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/proto/marshal/collections/repeated.py", line 125, in __getitem__
    return self._marshal.to_python(self._pb_type, self.pb[key])
                                                  ~~~~~~~^^^^^
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/litellm/utils.py", line 10218, in __next__
    response: Optional[ModelResponse] = self.chunk_creator(chunk=chunk)
                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/litellm/utils.py", line 10149, in chunk_creator
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/litellm/utils.py", line 8703, in exception_type
    raise e
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/litellm/utils.py", line 8678, in exception_type
    raise APIConnectionError(
litellm.exceptions.APIConnectionError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/userr/.local/bin/aider", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/aider/main.py", line 700, in main
    coder.run()
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/aider/coders/base_coder.py", line 397, in run
    new_user_message = self.run_loop()
                       ^^^^^^^^^^^^^^^
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/aider/coders/base_coder.py", line 481, in run_loop
    return self.send_new_user_message(inp)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/aider/coders/base_coder.py", line 536, in send_new_user_message
    interrupted = self.send(messages, functions=self.functions)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/aider/coders/base_coder.py", line 660, in send
    self.show_send_output_stream(completion)
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/aider/coders/base_coder.py", line 743, in show_send_output_stream
    for chunk in completion:
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/litellm/utils.py", line 10255, in __next__
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/litellm/utils.py", line 8703, in exception_type
    raise e
  File "/home/userr/.local/pipx/venvs/aider-chat/lib/python3.11/site-packages/litellm/utils.py", line 8671, in exception_type
    raise APIConnectionError(
litellm.exceptions.APIConnectionError: list index out of ra
nge

Twitter / LinkedIn details

No response

[krrishdholakia]

Hi @paul-gauthier default to retry. APIConnectionError has status code 500, which litellm._should_retry(status_code) maps to a True (reimplementation of openai's internal logic).

For safety, if an exception has a status code, you can just check it with litellm._should_retry to see if it's one you should retry for.

I'll also work on exception mapping this particular error b/c i've seen it on gemini and vertex, and retrying typically fixes it.

litellm/litellm/utils.py

Line 7642 in 5e6b6c4

def _should_retry(status_code: int):

[krrishdholakia]

Added exception mapping for this error and updated our docs with 'should_retry' logic as well - f7eee60