configuring custom model provider(which iscompatible with openAI) with `api_base` still try to use openaiAPI

[zigerZZZ]

Hello there, for some reason, openai and other model provider is lagging in my place. So I try to using custom model provider.Here's my code

from paperqa import Settings, ask
from paperqa.settings import AgentSettings,ParsingSettings
import os

OPENAI_API_KEY = '***'

model = 'openai/qwen-plus'
embedding_model = "openai/text-embedding-v4"

os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY

llm_config={
    "model_list": [{
        "model_name": model,
        "litellm_params": {
            "model": model,
            "api_base": "https://dashscope.aliyuncs.com/compatible-mode/v1",
            "api_key": OPENAI_API_KEY
        }
    }]
}

embedding_config={
    "model_list": [{
        "model_name": embedding_model,
        "litellm_params": {
            "model": embedding_model,
            "api_base": "https://dashscope.aliyuncs.com/compatible-mode/v1",
            "api_key": OPENAI_API_KEY
        }
    }]
}

answer_response = ask(
    "What is Paper-QA?",
    settings=Settings(
        llm=model, 
        llm_config=llm_config,
        summary_llm=model, 
        summary_llm_config=llm_config,
        agent=AgentSettings(agent_llm=model,
        agent_llm_config=llm_config),
        paper_directory="examples",
        embedding=embedding_model,
        embedding_config=embedding_config,
        parsing=ParsingSettings(use_doc_details=False)
    ),
)
print("Finished!")

Here's how it happens

[17:10:03] Error parsing PaperQA2.pdf, skipping index for this file.                                               
           ╭───────────────────────────────── Traceback (most recent call last) ──────────────────────────────────╮
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/aiohttp/connector. │
           │ py:1181 in _wrap_create_connection                                                                   │
           │                                                                                                      │
           │   1178 │   │   │   async with ceil_timeout(                                                          │
           │   1179 │   │   │   │   timeout.sock_connect, ceil_threshold=timeout.ceil_threshold                   │
           │   1180 │   │   │   ):                                                                                │
           │ ❱ 1181 │   │   │   │   sock = await aiohappyeyeballs.start_connection(                               │
           │   1182 │   │   │   │   │   addr_infos=addr_infos,                                                    │
           │   1183 │   │   │   │   │   local_addr_infos=self._local_addr_infos,                                  │
           │   1184 │   │   │   │   │   happy_eyeballs_delay=self._happy_eyeballs_delay,                          │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/aiohappyeyeballs/i │
           │ mpl.py:149 in start_connection                                                                       │
           │                                                                                                      │
           │   146 │   │   │   │   # We have a mix of OSError and RuntimeError                                    │
           │   147 │   │   │   │   # so we have to pick which one to raise.                                       │
           │   148 │   │   │   │   # and we raise OSError for compatibility                                       │
           │ ❱ 149 │   │   │   │   raise OSError(msg)                                                             │
           │   150 │   │   finally:                                                                               │
           │   151 │   │   │   all_exceptions = None  # type: ignore[assignment]                                  │
           │   152 │   │   │   exceptions = None  # type: ignore[assignment]                                      │
           ╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
           OSError: Multiple exceptions: [Errno 110] Connect call failed ('128.242.240.91', 443), [Errno 101]      
           Network is unreachable                                                                                  
                                                                                                                   
           The above exception was the direct cause of the following exception:                                    
                                                                                                                   
           ╭───────────────────────────────── Traceback (most recent call last) ──────────────────────────────────╮
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/custo │
           │ m_httpx/aiohttp_transport.py:55 in map_aiohttp_exceptions                                            │
           │                                                                                                      │
           │    52 @contextlib.contextmanager                                                                     │
           │    53 def map_aiohttp_exceptions() -> typing.Iterator[None]:                                         │
           │    54 │   try:                                                                                       │
           │ ❱  55 │   │   yield                                                                                  │
           │    56 │   except Exception as exc:                                                                   │
           │    57 │   │   mapped_exc = None                                                                      │
           │    58                                                                                                │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/custo │
           │ m_httpx/aiohttp_transport.py:207 in handle_async_request                                             │
           │                                                                                                      │
           │   204 │   │   │   │   data = request.stream  # type: ignore                                          │
           │   205 │   │   │   │   request.headers.pop("transfer-encoding", None)  # handled by aiohttp           │
           │   206 │   │   │                                                                                      │
           │ ❱ 207 │   │   │   response = await client_session.request(                                           │
           │   208 │   │   │   │   method=request.method,                                                         │
           │   209 │   │   │   │   url=YarlURL(str(request.url), encoded=True),                                   │
           │   210 │   │   │   │   headers=request.headers,                                                       │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/aiohttp/client.py: │
           │ 1473 in __aenter__                                                                                   │
           │                                                                                                      │
           │   1470 │   │   return self.__await__()                                                               │
           │   1471 │                                                                                             │
           │   1472 │   async def __aenter__(self) -> _RetType:                                                   │
           │ ❱ 1473 │   │   self._resp: _RetType = await self._coro                                               │
           │   1474 │   │   return await self._resp.__aenter__()                                                  │
           │   1475 │                                                                                             │
           │   1476 │   async def __aexit__(                                                                      │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/aiohttp/client.py: │
           │ 761 in _request                                                                                      │
           │                                                                                                      │
           │    758 │   │   │   │   │   │   handler = _connect_and_send_request                                   │
           │    759 │   │   │   │   │                                                                             │
           │    760 │   │   │   │   │   try:                                                                      │
           │ ❱  761 │   │   │   │   │   │   resp = await handler(req)                                             │
           │    762 │   │   │   │   │   # Client connector errors should not be retried                           │
           │    763 │   │   │   │   │   except (                                                                  │
           │    764 │   │   │   │   │   │   ConnectionTimeoutError,                                               │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/aiohttp/client.py: │
           │ 716 in _connect_and_send_request                                                                     │
           │                                                                                                      │
           │    713 │   │   │   │   │   │   # connection timeout                                                  │
           │    714 │   │   │   │   │   │   assert self._connector is not None                                    │
           │    715 │   │   │   │   │   │   try:                                                                  │
           │ ❱  716 │   │   │   │   │   │   │   conn = await self._connector.connect(                             │
           │    717 │   │   │   │   │   │   │   │   req, traces=traces, timeout=real_timeout                      │
           │    718 │   │   │   │   │   │   │   )                                                                 │
           │    719 │   │   │   │   │   │   except asyncio.TimeoutError as exc:                                   │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/aiohttp/connector. │
           │ py:598 in connect                                                                                    │
           │                                                                                                      │
           │    595 │   │   │   │   if traces:                                                                    │
           │    596 │   │   │   │   │   for trace in traces:                                                      │
           │    597 │   │   │   │   │   │   await trace.send_connection_create_start()                            │
           │ ❱  598 │   │   │   │   proto = await self._create_connection(req, traces, timeout)                   │
           │    599 │   │   │   │   if traces:                                                                    │
           │    600 │   │   │   │   │   for trace in traces:                                                      │
           │    601 │   │   │   │   │   │   await trace.send_connection_create_end()                              │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/aiohttp/connector. │
           │ py:1122 in _create_connection                                                                        │
           │                                                                                                      │
           │   1119 │   │   if req.proxy:                                                                         │
           │   1120 │   │   │   _, proto = await self._create_proxy_connection(req, traces, timeout)              │
           │   1121 │   │   else:                                                                                 │
           │ ❱ 1122 │   │   │   _, proto = await self._create_direct_connection(req, traces, timeout)             │
           │   1123 │   │                                                                                         │
           │   1124 │   │   return proto                                                                          │
           │   1125                                                                                               │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/aiohttp/connector. │
           │ py:1494 in _create_direct_connection                                                                 │
           │                                                                                                      │
           │   1491 │   │   │   return transp, proto                                                              │
           │   1492 │   │   else:                                                                                 │
           │   1493 │   │   │   assert last_exc is not None                                                       │
           │ ❱ 1494 │   │   │   raise last_exc                                                                    │
           │   1495 │                                                                                             │
           │   1496 │   async def _create_proxy_connection(                                                       │
           │   1497 │   │   self, req: ClientRequest, traces: List["Trace"], timeout: "ClientTimeout"             │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/aiohttp/connector. │
           │ py:1463 in _create_direct_connection                                                                 │
           │                                                                                                      │
           │   1460 │   │   │   )                                                                                 │
           │   1461 │   │   │                                                                                     │
           │   1462 │   │   │   try:                                                                              │
           │ ❱ 1463 │   │   │   │   transp, proto = await self._wrap_create_connection(                           │
           │   1464 │   │   │   │   │   self._factory,                                                            │
           │   1465 │   │   │   │   │   timeout=timeout,                                                          │
           │   1466 │   │   │   │   │   ssl=sslcontext,                                                           │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/aiohttp/connector. │
           │ py:1204 in _wrap_create_connection                                                                   │
           │                                                                                                      │
           │   1201 │   │   except OSError as exc:                                                                │
           │   1202 │   │   │   if exc.errno is None and isinstance(exc, asyncio.TimeoutError):                   │
           │   1203 │   │   │   │   raise                                                                         │
           │ ❱ 1204 │   │   │   raise client_error(req.connection_key, exc) from exc                              │
           │   1205 │                                                                                             │
           │   1206 │   async def _wrap_existing_connection(                                                      │
           │   1207 │   │   self,                                                                                 │
           ╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
           ClientConnectorError: Cannot connect to host api.openai.com:443 ssl:default [None]                      
                                                                                                                   
           The above exception was the direct cause of the following exception:                                    
                                                                                                                   
           ╭───────────────────────────────── Traceback (most recent call last) ──────────────────────────────────╮
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/_base_clien │
           │ t.py:1490 in request                                                                                 │
           │                                                                                                      │
           │   1487 │   │   │                                                                                     │
           │   1488 │   │   │   response = None                                                                   │
           │   1489 │   │   │   try:                                                                              │
           │ ❱ 1490 │   │   │   │   response = await self._client.send(                                           │
           │   1491 │   │   │   │   │   request,                                                                  │
           │   1492 │   │   │   │   │   stream=stream or self._should_stream_response_body(request=request),      │
           │   1493 │   │   │   │   │   **kwargs,                                                                 │
           │                                                                                                      │
           │ /fastone/users/shiny/.local/lib/python3.11/site-packages/httpx/_client.py:1629 in send               │
           │                                                                                                      │
           │   1626 │   │                                                                                         │
           │   1627 │   │   auth = self._build_request_auth(request, auth)                                        │
           │   1628 │   │                                                                                         │
           │ ❱ 1629 │   │   response = await self._send_handling_auth(                                            │
           │   1630 │   │   │   request,                                                                          │
           │   1631 │   │   │   auth=auth,                                                                        │
           │   1632 │   │   │   follow_redirects=follow_redirects,                                                │
           │                                                                                                      │
           │ /fastone/users/shiny/.local/lib/python3.11/site-packages/httpx/_client.py:1657 in                    │
           │ _send_handling_auth                                                                                  │
           │                                                                                                      │
           │   1654 │   │   │   request = await auth_flow.__anext__()                                             │
           │   1655 │   │   │                                                                                     │
           │   1656 │   │   │   while True:                                                                       │
           │ ❱ 1657 │   │   │   │   response = await self._send_handling_redirects(                               │
           │   1658 │   │   │   │   │   request,                                                                  │
           │   1659 │   │   │   │   │   follow_redirects=follow_redirects,                                        │
           │   1660 │   │   │   │   │   history=history,                                                          │
           │                                                                                                      │
           │ /fastone/users/shiny/.local/lib/python3.11/site-packages/httpx/_client.py:1694 in                    │
           │ _send_handling_redirects                                                                             │
           │                                                                                                      │
           │   1691 │   │   │   for hook in self._event_hooks["request"]:                                         │
           │   1692 │   │   │   │   await hook(request)                                                           │
           │   1693 │   │   │                                                                                     │
           │ ❱ 1694 │   │   │   response = await self._send_single_request(request)                               │
           │   1695 │   │   │   try:                                                                              │
           │   1696 │   │   │   │   for hook in self._event_hooks["response"]:                                    │
           │   1697 │   │   │   │   │   await hook(response)                                                      │
           │                                                                                                      │
           │ /fastone/users/shiny/.local/lib/python3.11/site-packages/httpx/_client.py:1730 in                    │
           │ _send_single_request                                                                                 │
           │                                                                                                      │
           │   1727 │   │   │   )                                                                                 │
           │   1728 │   │                                                                                         │
           │   1729 │   │   with request_context(request=request):                                                │
           │ ❱ 1730 │   │   │   response = await transport.handle_async_request(request)                          │
           │   1731 │   │                                                                                         │
           │   1732 │   │   assert isinstance(response.stream, AsyncByteStream)                                   │
           │   1733 │   │   response.request = request                                                            │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/custo │
           │ m_httpx/aiohttp_transport.py:200 in handle_async_request                                             │
           │                                                                                                      │
           │   197 │   │   # Use helper to ensure we have a valid session for the current event loop              │
           │   198 │   │   client_session = self._get_valid_client_session()                                      │
           │   199 │   │                                                                                          │
           │ ❱ 200 │   │   with map_aiohttp_exceptions():                                                         │
           │   201 │   │   │   try:                                                                               │
           │   202 │   │   │   │   data = request.content                                                         │
           │   203 │   │   │   except httpx.RequestNotRead:                                                       │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/contextlib.py:158 in __exit__    │
           │                                                                                                      │
           │   155 │   │   │   │   # tell if we get the same exception back                                       │
           │   156 │   │   │   │   value = typ()                                                                  │
           │   157 │   │   │   try:                                                                               │
           │ ❱ 158 │   │   │   │   self.gen.throw(typ, value, traceback)                                          │
           │   159 │   │   │   except StopIteration as exc:                                                       │
           │   160 │   │   │   │   # Suppress StopIteration *unless* it's the same exception that                 │
           │   161 │   │   │   │   # was passed to throw().  This prevents a StopIteration                        │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/custo │
           │ m_httpx/aiohttp_transport.py:69 in map_aiohttp_exceptions                                            │
           │                                                                                                      │
           │    66 │   │   │   raise                                                                              │
           │    67 │   │                                                                                          │
           │    68 │   │   message = str(exc)                                                                     │
           │ ❱  69 │   │   raise mapped_exc(message) from exc                                                     │
           │    70                                                                                                │
           │    71                                                                                                │
           │    72 class AiohttpResponseStream(httpx.AsyncByteStream):                                            │
           ╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
           ConnectError: Cannot connect to host api.openai.com:443 ssl:default [None]                              
                                                                                                                   
           The above exception was the direct cause of the following exception:                                    
                                                                                                                   
           ╭───────────────────────────────── Traceback (most recent call last) ──────────────────────────────────╮
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
           │ i/openai.py:1116 in aembedding                                                                       │
           │                                                                                                      │
           │   1113 │   │   │   │   max_retries=max_retries,                                                      │
           │   1114 │   │   │   │   client=client,                                                                │
           │   1115 │   │   │   )                                                                                 │
           │ ❱ 1116 │   │   │   headers, response = await self.make_openai_embedding_request(                     │
           │   1117 │   │   │   │   openai_aclient=openai_aclient,                                                │
           │   1118 │   │   │   │   data=data,                                                                    │
           │   1119 │   │   │   │   timeout=timeout,                                                              │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/litellm_co │
           │ re_utils/logging_utils.py:135 in async_wrapper                                                       │
           │                                                                                                      │
           │   132 │   │   async def async_wrapper(*args, **kwargs):                                              │
           │   133 │   │   │   start_time = datetime.now()                                                        │
           │   134 │   │   │   try:                                                                               │
           │ ❱ 135 │   │   │   │   result = await func(*args, **kwargs)                                           │
           │   136 │   │   │   │   return result                                                                  │
           │   137 │   │   │   finally:                                                                           │
           │   138 │   │   │   │   end_time = datetime.now()                                                      │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
           │ i/openai.py:1069 in make_openai_embedding_request                                                    │
           │                                                                                                      │
           │   1066 │   │   │   response = raw_response.parse()                                                   │
           │   1067 │   │   │   return headers, response                                                          │
           │   1068 │   │   except Exception as e:                                                                │
           │ ❱ 1069 │   │   │   raise e                                                                           │
           │   1070 │                                                                                             │
           │   1071 │   @track_llm_api_timing()                                                                   │
           │   1072 │   def make_sync_openai_embedding_request(                                                   │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
           │ i/openai.py:1062 in make_openai_embedding_request                                                    │
           │                                                                                                      │
           │   1059 │   │   - call embeddings.create by default                                                   │
           │   1060 │   │   """                                                                                   │
           │   1061 │   │   try:                                                                                  │
           │ ❱ 1062 │   │   │   raw_response = await openai_aclient.embeddings.with_raw_response.create(          │
           │   1063 │   │   │   │   **data, timeout=timeout                                                       │
           │   1064 │   │   │   )  # type: ignore                                                                 │
           │   1065 │   │   │   headers = dict(raw_response.headers)                                              │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/_legacy_res │
           │ ponse.py:381 in wrapped                                                                              │
           │                                                                                                      │
           │   378 │   │                                                                                          │
           │   379 │   │   kwargs["extra_headers"] = extra_headers                                                │
           │   380 │   │                                                                                          │
           │ ❱ 381 │   │   return cast(LegacyAPIResponse[R], await func(*args, **kwargs))                         │
           │   382 │                                                                                              │
           │   383 │   return wrapped                                                                             │
           │   384                                                                                                │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/resources/e │
           │ mbeddings.py:245 in create                                                                           │
           │                                                                                                      │
           │   242 │   │   │                                                                                      │
           │   243 │   │   │   return obj                                                                         │
           │   244 │   │                                                                                          │
           │ ❱ 245 │   │   return await self._post(                                                               │
           │   246 │   │   │   "/embeddings",                                                                     │
           │   247 │   │   │   body=maybe_transform(params, embedding_create_params.EmbeddingCreateParams),       │
           │   248 │   │   │   options=make_request_options(                                                      │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/_base_clien │
           │ t.py:1748 in post                                                                                    │
           │                                                                                                      │
           │   1745 │   │   opts = FinalRequestOptions.construct(                                                 │
           │   1746 │   │   │   method="post", url=path, json_data=body, files=await                              │
           │        async_to_httpx_files(files), **options                                                        │
           │   1747 │   │   )                                                                                     │
           │ ❱ 1748 │   │   return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)        │
           │   1749 │                                                                                             │
           │   1750 │   async def patch(                                                                          │
           │   1751 │   │   self,                                                                                 │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/_base_clien │
           │ t.py:1522 in request                                                                                 │
           │                                                                                                      │
           │   1519 │   │   │   │   │   continue                                                                  │
           │   1520 │   │   │   │                                                                                 │
           │   1521 │   │   │   │   log.debug("Raising connection error")                                         │
           │ ❱ 1522 │   │   │   │   raise APIConnectionError(request=request) from err                            │
           │   1523 │   │   │                                                                                     │
           │   1524 │   │   │   log.debug(                                                                        │
           │   1525 │   │   │   │   'HTTP Response: %s %s "%i %s" %s',                                            │
           ╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
           APIConnectionError: Connection error. 

I'm sure the API is compatible with openAI,because this code works

import os
from litellm import completion

os.environ["OPENAI_API_KEY"] = "***"
messages = [{ "content": "Hello from litellm!","role": "user"}]
response = completion(model="openai/qwen-plus", messages=messages, base_url="https://dashscope.aliyuncs.com/compatible-mode/v1")
print(response.choices[0].message.content)

>Hello! How can I assist you today? If you have any questions or need help with the `litellm` library, feel free to ask!

[dosubot]

Hi @zigerZZZ! I'm Dosu and I’m helping the paper-qa team.

Your configuration is almost correct, but the root cause is that paper-qa defaults to OpenAI endpoints for all four model slots: llm, summary_llm, agent_llm, and embedding. Even if you set the api_base for your LLM, if the embedding model is not explicitly set up with the correct provider and endpoint, paper-qa will still try to use OpenAI's API for embeddings, which matches the error traceback you see.

To fix this, make sure you set both the embedding and embedding_config in your Settings object, just like you did for the LLM. Your code already does this, but double-check that the embedding_model you specify is actually available from your custom provider at the endpoint you set, and that the model name matches what your provider expects. If your provider does not support the embedding model "openai/text-embedding-v4", paper-qa will fall back to OpenAI's API for embeddings, causing the connection error you see. This is a common pitfall and is discussed in similar issues: see here.

If your provider does not offer embeddings, you need to use an embedding model that is available from a provider you can reach, or use a local embedding model. If your provider does support embeddings, make sure the model name and endpoint are correct in both embedding and embedding_config.

In summary: the error is because paper-qa is still trying to use OpenAI for embeddings, not completions. Double-check your embedding model/provider compatibility and configuration. If you need to use a different embedding model, update the embedding and embedding_config fields accordingly.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[zigerZZZ]

Thx,@dosu.Hover, I have checked the embedding model which is available.Like this

import os
from litellm import embedding

os.environ["OPENAI_API_KEY"] = "***"
response = embedding(model="openai/text-embedding-v4", 
                    input="The quality of the clothes is excellent and very beautiful. It's worth the wait. I like them and will come here to buy them in the future",
                    encoding_format="float",
                    api_base="https://dashscope.aliyuncs.com/compatible-mode/v1")
print(response.model_dump_json())

>{"model":"text-embedding-v4","data":[{"embedding":[0.014462350867688656, ......

BTW, this model is not params dimensions=1024

response = embedding(model="openai/text-embedding-v4", 
                    input="The quality of the clothes is excellent and very beautiful. It's worth the wait. I like them and will come here to buy them in the future",
                    encoding_format="float",
                    dimensions=1024, # THIS WILL MAKE ERRO
                    api_base="https://dashscope.aliyuncs.com/compatible-mode/v1")
print(response.model_dump_json())

UnsupportedParamsError: litellm.UnsupportedParamsError: Setting dimensions is not supported for OpenAI `text-embedding-3` and later models. To drop it from the call, set `litellm.drop_params = True`

[jamesbraza]

Hi @zigerZZZ, I think the issue may have been resolved in a dependency of PaperQA called lmi in Future-House/ldp#280

Can you pip install fhlmi>=0.28 and let us know if you still hit the same issue?

[zigerZZZ]

thx! but another problem occurs, it's BadRequestError saying that batch size is invalid, it should not be larger than 10
Shall I change to any other LLM?

[09:55:22] Error parsing PaperQA2.pdf, skipping index for this file.                                               
           ╭───────────────────────────────── Traceback (most recent call last) ──────────────────────────────────╮
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
           │ i/openai.py:1116 in aembedding                                                                       │
           │                                                                                                      │
           │   1113 │   │   │   │   max_retries=max_retries,                                                      │
           │   1114 │   │   │   │   client=client,                                                                │
           │   1115 │   │   │   )                                                                                 │
           │ ❱ 1116 │   │   │   headers, response = await self.make_openai_embedding_request(                     │
           │   1117 │   │   │   │   openai_aclient=openai_aclient,                                                │
           │   1118 │   │   │   │   data=data,                                                                    │
           │   1119 │   │   │   │   timeout=timeout,                                                              │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/litellm_co │
           │ re_utils/logging_utils.py:135 in async_wrapper                                                       │
           │                                                                                                      │
           │   132 │   │   async def async_wrapper(*args, **kwargs):                                              │
           │   133 │   │   │   start_time = datetime.now()                                                        │
           │   134 │   │   │   try:                                                                               │
           │ ❱ 135 │   │   │   │   result = await func(*args, **kwargs)                                           │
           │   136 │   │   │   │   return result                                                                  │
           │   137 │   │   │   finally:                                                                           │
           │   138 │   │   │   │   end_time = datetime.now()                                                      │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
           │ i/openai.py:1069 in make_openai_embedding_request                                                    │
           │                                                                                                      │
           │   1066 │   │   │   response = raw_response.parse()                                                   │
           │   1067 │   │   │   return headers, response                                                          │
           │   1068 │   │   except Exception as e:                                                                │
           │ ❱ 1069 │   │   │   raise e                                                                           │
           │   1070 │                                                                                             │
           │   1071 │   @track_llm_api_timing()                                                                   │
           │   1072 │   def make_sync_openai_embedding_request(                                                   │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
           │ i/openai.py:1062 in make_openai_embedding_request                                                    │
           │                                                                                                      │
           │   1059 │   │   - call embeddings.create by default                                                   │
           │   1060 │   │   """                                                                                   │
           │   1061 │   │   try:                                                                                  │
           │ ❱ 1062 │   │   │   raw_response = await openai_aclient.embeddings.with_raw_response.create(          │
           │   1063 │   │   │   │   **data, timeout=timeout                                                       │
           │   1064 │   │   │   )  # type: ignore                                                                 │
           │   1065 │   │   │   headers = dict(raw_response.headers)                                              │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/_legacy_res │
           │ ponse.py:381 in wrapped                                                                              │
           │                                                                                                      │
           │   378 │   │                                                                                          │
           │   379 │   │   kwargs["extra_headers"] = extra_headers                                                │
           │   380 │   │                                                                                          │
           │ ❱ 381 │   │   return cast(LegacyAPIResponse[R], await func(*args, **kwargs))                         │
           │   382 │                                                                                              │
           │   383 │   return wrapped                                                                             │
           │   384                                                                                                │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/resources/e │
           │ mbeddings.py:245 in create                                                                           │
           │                                                                                                      │
           │   242 │   │   │                                                                                      │
           │   243 │   │   │   return obj                                                                         │
           │   244 │   │                                                                                          │
           │ ❱ 245 │   │   return await self._post(                                                               │
           │   246 │   │   │   "/embeddings",                                                                     │
           │   247 │   │   │   body=maybe_transform(params, embedding_create_params.EmbeddingCreateParams),       │
           │   248 │   │   │   options=make_request_options(                                                      │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/_base_clien │
           │ t.py:1748 in post                                                                                    │
           │                                                                                                      │
           │   1745 │   │   opts = FinalRequestOptions.construct(                                                 │
           │   1746 │   │   │   method="post", url=path, json_data=body, files=await                              │
           │        async_to_httpx_files(files), **options                                                        │
           │   1747 │   │   )                                                                                     │
           │ ❱ 1748 │   │   return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)        │
           │   1749 │                                                                                             │
           │   1750 │   async def patch(                                                                          │
           │   1751 │   │   self,                                                                                 │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/_base_clien │
           │ t.py:1555 in request                                                                                 │
           │                                                                                                      │
           │   1552 │   │   │   │   │   await err.response.aread()                                                │
           │   1553 │   │   │   │                                                                                 │
           │   1554 │   │   │   │   log.debug("Re-raising status error")                                          │
           │ ❱ 1555 │   │   │   │   raise self._make_status_error_from_response(err.response) from None           │
           │   1556 │   │   │                                                                                     │
           │   1557 │   │   │   break                                                                             │
           │   1558                                                                                               │
           ╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
           BadRequestError: Error code: 400 - {'error': {'code': 'InvalidParameter', 'param': None, 'message':     
           '<400> InternalError.Algo.InvalidParameter: Value error, batch size is invalid, it should not be larger 
           than 10.: input.contents', 'type': 'InvalidParameter'}, 'id': '5aab869a-0505-9449-9a2e-53f6a0f861d0',   
           'request_id': '5aab869a-0505-9449-9a2e-53f6a0f861d0'}                                                   
                                                                                                                   
           During handling of the above exception, another exception occurred:                                     
                                                                                                                   
           ╭───────────────────────────────── Traceback (most recent call last) ──────────────────────────────────╮
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/main.py:33 │
           │ 74 in aembedding                                                                                     │
           │                                                                                                      │
           │   3371 │   │   elif isinstance(init_response, EmbeddingResponse):  ## CACHING SCENARIO               │
           │   3372 │   │   │   response = init_response                                                          │
           │   3373 │   │   elif asyncio.iscoroutine(init_response):                                              │
           │ ❱ 3374 │   │   │   response = await init_response  # type: ignore                                    │
           │   3375 │   │   if (                                                                                  │
           │   3376 │   │   │   response is not None                                                              │
           │   3377 │   │   │   and isinstance(response, EmbeddingResponse)                                       │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
           │ i/openai.py:1161 in aembedding                                                                       │
           │                                                                                                      │
           │   1158 │   │   │   error_response = getattr(e, "response", None)                                     │
           │   1159 │   │   │   if error_headers is None and error_response:                                      │
           │   1160 │   │   │   │   error_headers = getattr(error_response, "headers", None)                      │
           │ ❱ 1161 │   │   │   raise OpenAIError(                                                                │
           │   1162 │   │   │   │   status_code=status_code, message=error_text, headers=error_headers            │
           │   1163 │   │   │   )                                                                                 │
           │   1164                                                                                               │
           ╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
           OpenAIError: Error code: 400 - {'error': {'code': 'InvalidParameter', 'param': None, 'message': '<400>  
           InternalError.Algo.InvalidParameter: Value error, batch size is invalid, it should not be larger than   
           10.: input.contents', 'type': 'InvalidParameter'}, 'id': '5aab869a-0505-9449-9a2e-53f6a0f861d0',        
           'request_id': '5aab869a-0505-9449-9a2e-53f6a0f861d0'}                                                   
                                                                                                                   
           During handling of the above exception, another exception occurred:                                     
                                                                                                                   
           ╭───────────────────────────────── Traceback (most recent call last) ──────────────────────────────────╮
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/paperqa/agents/sea │
           │ rch.py:528 in process_file                                                                           │
           │                                                                                                      │
           │   525 │   │   │                                                                                      │
           │   526 │   │   │   tmp_docs = Docs()                                                                  │
           │   527 │   │   │   try:                                                                               │
           │ ❱ 528 │   │   │   │   await tmp_docs.aadd(                                                           │
           │   529 │   │   │   │   │   path=abs_file_path,                                                        │
           │   530 │   │   │   │   │   fields=["title", "author", "journal", "year"],                             │
           │   531 │   │   │   │   │   settings=settings,                                                         │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/paperqa/docs.py:42 │
           │ 3 in aadd                                                                                            │
           │                                                                                                      │
           │   420 │   │   │   │   f"This does not look like a text document: {path}. Pass disable_check"         │
           │   421 │   │   │   │   " to ignore this error."                                                       │
           │   422 │   │   │   )                                                                                  │
           │ ❱ 423 │   │   if await self.aadd_texts(texts, doc, all_settings, embedding_model):                   │
           │   424 │   │   │   return docname                                                                     │
           │   425 │   │   return None                                                                            │
           │   426                                                                                                │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/paperqa/docs.py:48 │
           │ 1 in aadd_texts                                                                                      │
           │                                                                                                      │
           │   478 │   │   if embedding_model and texts[0].embedding is None:                                     │
           │   479 │   │   │   for t, t_embedding in zip(                                                         │
           │   480 │   │   │   │   texts,                                                                         │
           │ ❱ 481 │   │   │   │   await embedding_model.embed_documents(texts=[t.text for t in texts]),          │
           │   482 │   │   │   │   strict=True,                                                                   │
           │   483 │   │   │   ):                                                                                 │
           │   484 │   │   │   │   t.embedding = t_embedding                                                      │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/lmi/embeddings.py: │
           │ 166 in embed_documents                                                                               │
           │                                                                                                      │
           │   163 │   │   │   │   )                                                                              │
           │   164 │   │   │   )                                                                                  │
           │   165 │   │   │                                                                                      │
           │ ❱ 166 │   │   │   response = await track_costs(self.router.aembedding)(                              │
           │   167 │   │   │   │   model=self.name,                                                               │
           │   168 │   │   │   │   input=texts[i : i + batch_size],                                               │
           │   169 │   │   │   │   dimensions=self.ndim,                                                          │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/lmi/cost_tracker.p │
           │ y:91 in wrapped_func                                                                                 │
           │                                                                                                      │
           │    88 │   """                                                                                        │
           │    89 │                                                                                              │
           │    90 │   async def wrapped_func(*args, **kwargs):                                                   │
           │ ❱  91 │   │   response = await func(*args, **kwargs)                                                 │
           │    92 │   │   if GLOBAL_COST_TRACKER.enabled.get():                                                  │
           │    93 │   │   │   GLOBAL_COST_TRACKER.record(response)                                               │
           │    94 │   │   return response                                                                        │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
           │ 2679 in aembedding                                                                                   │
           │                                                                                                      │
           │   2676 │   │   │   │   │   original_exception=e,                                                     │
           │   2677 │   │   │   │   )                                                                             │
           │   2678 │   │   │   )                                                                                 │
           │ ❱ 2679 │   │   │   raise e                                                                           │
           │   2680 │                                                                                             │
           │   2681 │   async def _aembedding(self, input: Union[str, List], model: str, **kwargs):               │
           │   2682 │   │   model_name = None                                                                     │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
           │ 2668 in aembedding                                                                                   │
           │                                                                                                      │
           │   2665 │   │   │   kwargs["input"] = input                                                           │
           │   2666 │   │   │   kwargs["original_function"] = self._aembedding                                    │
           │   2667 │   │   │   self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)                  │
           │ ❱ 2668 │   │   │   response = await self.async_function_with_fallbacks(**kwargs)                     │
           │   2669 │   │   │   return response                                                                   │
           │   2670 │   │   except Exception as e:                                                                │
           │   2671 │   │   │   asyncio.create_task(                                                              │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
           │ 3542 in async_function_with_fallbacks                                                                │
           │                                                                                                      │
           │   3539 │   │   │   │   │   │   )                                                                     │
           │   3540 │   │   │   │   │   )                                                                         │
           │   3541 │   │   │                                                                                     │
           │ ❱ 3542 │   │   │   raise original_exception                                                          │
           │   3543 │                                                                                             │
           │   3544 │   def _handle_mock_testing_fallbacks(                                                       │
           │   3545 │   │   self,                                                                                 │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
           │ 3356 in async_function_with_fallbacks                                                                │
           │                                                                                                      │
           │   3353 │   │   │   │   │   *args, **kwargs, mock_timeout=mock_timeout                                │
           │   3354 │   │   │   │   )                                                                             │
           │   3355 │   │   │   else:                                                                             │
           │ ❱ 3356 │   │   │   │   response = await self.async_function_with_retries(*args, **kwargs)            │
           │   3357 │   │   │   verbose_router_logger.debug(f"Async Response: {response}")                        │
           │   3358 │   │   │   response = add_fallback_headers_to_response(                                      │
           │   3359 │   │   │   │   response=response,                                                            │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
           │ 3734 in async_function_with_retries                                                                  │
           │                                                                                                      │
           │   3731 │   │   │   │   setattr(original_exception, "max_retries", num_retries)                       │
           │   3732 │   │   │   │   setattr(original_exception, "num_retries", current_attempt)                   │
           │   3733 │   │   │                                                                                     │
           │ ❱ 3734 │   │   │   raise original_exception                                                          │
           │   3735 │                                                                                             │
           │   3736 │   async def make_call(self, original_function: Any, *args, **kwargs):                       │
           │   3737 │   │   """                                                                                   │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
           │ 3625 in async_function_with_retries                                                                  │
           │                                                                                                      │
           │   3622 │   │   │   │   model_group=model_group, kwargs=kwargs                                        │
           │   3623 │   │   │   )                                                                                 │
           │   3624 │   │   │   # if the function call is successful, no exception will be raised and we'll       │
           │        break out of the loop                                                                         │
           │ ❱ 3625 │   │   │   response = await self.make_call(original_function, *args, **kwargs)               │
           │   3626 │   │   │   response = add_retry_headers_to_response(                                         │
           │   3627 │   │   │   │   response=response, attempted_retries=0, max_retries=None                      │
           │   3628 │   │   │   )                                                                                 │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
           │ 3743 in make_call                                                                                    │
           │                                                                                                      │
           │   3740 │   │   model_group = kwargs.get("model")                                                     │
           │   3741 │   │   response = original_function(*args, **kwargs)                                         │
           │   3742 │   │   if inspect.iscoroutinefunction(response) or inspect.isawaitable(response):            │
           │ ❱ 3743 │   │   │   response = await response                                                         │
           │   3744 │   │   ## PROCESS RESPONSE HEADERS                                                           │
           │   3745 │   │   response = await self.set_response_headers(                                           │
           │   3746 │   │   │   response=response, model_group=model_group                                        │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
           │ 2749 in _aembedding                                                                                  │
           │                                                                                                      │
           │   2746 │   │   │   )                                                                                 │
           │   2747 │   │   │   if model_name is not None:                                                        │
           │   2748 │   │   │   │   self.fail_calls[model_name] += 1                                              │
           │ ❱ 2749 │   │   │   raise e                                                                           │
           │   2750 │                                                                                             │
           │   2751 │   #### FILES API ####                                                                       │
           │   2752 │   async def acreate_file(                                                                   │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
           │ 2736 in _aembedding                                                                                  │
           │                                                                                                      │
           │   2733 │   │   │   │   await self.async_routing_strategy_pre_call_checks(                            │
           │   2734 │   │   │   │   │   deployment=deployment, parent_otel_span=parent_otel_span                  │
           │   2735 │   │   │   │   )                                                                             │
           │ ❱ 2736 │   │   │   │   response = await response                                                     │
           │   2737 │   │   │                                                                                     │
           │   2738 │   │   │   self.success_calls[model_name] += 1                                               │
           │   2739 │   │   │   verbose_router_logger.info(                                                       │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/utils.py:1 │
           │ 494 in wrapper_async                                                                                 │
           │                                                                                                      │
           │   1491 │   │   │                                                                                     │
           │   1492 │   │   │   timeout = _get_wrapper_timeout(kwargs=kwargs, exception=e)                        │
           │   1493 │   │   │   setattr(e, "timeout", timeout)                                                    │
           │ ❱ 1494 │   │   │   raise e                                                                           │
           │   1495 │                                                                                             │
           │   1496 │   is_coroutine = inspect.iscoroutinefunction(original_function)                             │
           │   1497                                                                                               │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/utils.py:1 │
           │ 355 in wrapper_async                                                                                 │
           │                                                                                                      │
           │   1352 │   │   │   │   return _caching_handler_response.final_embedding_cached_response              │
           │   1353 │   │   │                                                                                     │
           │   1354 │   │   │   # MODEL CALL                                                                      │
           │ ❱ 1355 │   │   │   result = await original_function(*args, **kwargs)                                 │
           │   1356 │   │   │   end_time = datetime.datetime.now()                                                │
           │   1357 │   │   │   if "stream" in kwargs and kwargs["stream"] is True:                               │
           │   1358 │   │   │   │   if (                                                                          │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/main.py:33 │
           │ 89 in aembedding                                                                                     │
           │                                                                                                      │
           │   3386 │   │   return response                                                                       │
           │   3387 │   except Exception as e:                                                                    │
           │   3388 │   │   custom_llm_provider = custom_llm_provider or "openai"                                 │
           │ ❱ 3389 │   │   raise exception_type(                                                                 │
           │   3390 │   │   │   model=model,                                                                      │
           │   3391 │   │   │   custom_llm_provider=custom_llm_provider,                                          │
           │   3392 │   │   │   original_exception=e,                                                             │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/litellm_co │
           │ re_utils/exception_mapping_utils.py:2239 in exception_type                                           │
           │                                                                                                      │
           │   2236 │   │   # don't let an error with mapping interrupt the user from receiving an error          │
           │        from the llm api calls                                                                        │
           │   2237 │   │   if exception_mapping_worked:                                                          │
           │   2238 │   │   │   setattr(e, "litellm_response_headers", litellm_response_headers)                  │
           │ ❱ 2239 │   │   │   raise e                                                                           │
           │   2240 │   │   else:                                                                                 │
           │   2241 │   │   │   for error_type in litellm.LITELLM_EXCEPTION_TYPES:                                │
           │   2242 │   │   │   │   if isinstance(e, error_type):                                                 │
           │                                                                                                      │
           │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/litellm_co │
           │ re_utils/exception_mapping_utils.py:402 in exception_type                                            │
           │                                                                                                      │
           │    399 │   │   │   │   │   exception_mapping_worked = True                                           │
           │    400 │   │   │   │   │   if original_exception.status_code == 400:                                 │
           │    401 │   │   │   │   │   │   exception_mapping_worked = True                                       │
           │ ❱  402 │   │   │   │   │   │   raise BadRequestError(                                                │
           │    403 │   │   │   │   │   │   │   message=f"{exception_provider} - {message}",                      │
           │    404 │   │   │   │   │   │   │   llm_provider=custom_llm_provider,                                 │
           │    405 │   │   │   │   │   │   │   model=model,                                                      │
           ╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
           BadRequestError: litellm.BadRequestError: OpenAIException - Error code: 400 - {'error': {'code':        
           'InvalidParameter', 'param': None, 'message': '<400> InternalError.Algo.InvalidParameter: Value error,  
           batch size is invalid, it should not be larger than 10.: input.contents', 'type': 'InvalidParameter'},  
           'id': '5aab869a-0505-9449-9a2e-53f6a0f861d0', 'request_id': '5aab869a-0505-9449-9a2e-53f6a0f861d0'}.    
           Received Model Group=openai/text-embedding-v4                                                           
           Available Model Group Fallbacks=None LiteLLM Retried: 2 times, LiteLLM Max Retries: 3                   

[Endorphins20]

if embedding_model and texts[0].embedding is None:
batch_size = 10 # Set batch size to avoid API errors
text_embeddings = []
for i in range(0, len(texts), batch_size):
batch = texts[i : i + batch_size]
batch_embeddings = await embedding_model.embed_documents(
texts=[t.text for t in batch]
)
text_embeddings.extend(batch_embeddings)
for t, t_embedding in zip(
texts,
text_embeddings,
strict=True,
):
t.embedding = t_embedding

replace paperqa\docs.py

thx! but another problem occurs, it's BadRequestError saying that batch size is invalid, it should not be larger than 10谢谢！但是发生了另一个问题，这是BadrequeStror，说批处理大小无效，它不应大于10 Shall I change to any other LLM?我可以更改为其他LLM吗？

[09:55:22] Error parsing PaperQA2.pdf, skipping index for this file.
╭───────────────────────────────── Traceback (most recent call last) ──────────────────────────────────╮
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
│ i/openai.py:1116 in aembedding │
│ │
│ 1113 │ │ │ │ max_retries=max_retries, │
│ 1114 │ │ │ │ client=client, │
│ 1115 │ │ │ ) │
│ ❱ 1116 │ │ │ headers, response = await self.make_openai_embedding_request( │
│ 1117 │ │ │ │ openai_aclient=openai_aclient, │
│ 1118 │ │ │ │ data=data, │
│ 1119 │ │ │ │ timeout=timeout, │
│ │
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/litellm_co │
│ re_utils/logging_utils.py:135 in async_wrapper │
│ │
│ 132 │ │ async def async_wrapper(*args, **kwargs): │
│ 133 │ │ │ start_time = datetime.now() │
│ 134 │ │ │ try: │
│ ❱ 135 │ │ │ │ result = await func(*args, **kwargs) │
│ 136 │ │ │ │ return result │
│ 137 │ │ │ finally: │
│ 138 │ │ │ │ end_time = datetime.now() │
│ │
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
│ i/openai.py:1069 in make_openai_embedding_request │
│ │
│ 1066 │ │ │ response = raw_response.parse() │
│ 1067 │ │ │ return headers, response │
│ 1068 │ │ except Exception as e: │
│ ❱ 1069 │ │ │ raise e │
│ 1070 │ │
│ 1071 │ @track_llm_api_timing() │
│ 1072 │ def make_sync_openai_embedding_request( │
│ │
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
│ i/openai.py:1062 in make_openai_embedding_request │
│ │
│ 1059 │ │ - call embeddings.create by default │
│ 1060 │ │ """ │
│ 1061 │ │ try: │
│ ❱ 1062 │ │ │ raw_response = await openai_aclient.embeddings.with_raw_response.create( │
│ 1063 │ │ │ │ **data, timeout=timeout │
│ 1064 │ │ │ ) # type: ignore │
│ 1065 │ │ │ headers = dict(raw_response.headers) │
│ │
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/_legacy_res │
│ ponse.py:381 in wrapped │
│ │
│ 378 │ │ │
│ 379 │ │ kwargs["extra_headers"] = extra_headers │
│ 380 │ │ │
│ ❱ 381 │ │ return cast(LegacyAPIResponse[R], await func(*args, **kwargs)) │
│ 382 │ │
│ 383 │ return wrapped │
│ 384 │
│ │
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/resources/e │
│ mbeddings.py:245 in create │
│ │
│ 242 │ │ │ │
│ 243 │ │ │ return obj │
│ 244 │ │ │
│ ❱ 245 │ │ return await self._post( │
│ 246 │ │ │ "/embeddings", │
│ 247 │ │ │ body=maybe_transform(params, embedding_create_params.EmbeddingCreateParams), │
│ 248 │ │ │ options=make_request_options( │
│ │
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/_base_clien │
│ t.py:1748 in post │
│ │
│ 1745 │ │ opts = FinalRequestOptions.construct( │
│ 1746 │ │ │ method="post", url=path, json_data=body, files=await │
│ async_to_httpx_files(files), **options │
│ 1747 │ │ ) │
│ ❱ 1748 │ │ return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls) │
│ 1749 │ │
│ 1750 │ async def patch( │
│ 1751 │ │ self, │
│ │
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/_base_clien │
│ t.py:1555 in request │
│ │
│ 1552 │ │ │ │ │ await err.response.aread() │
│ 1553 │ │ │ │ │
│ 1554 │ │ │ │ log.debug("Re-raising status error") │
│ ❱ 1555 │ │ │ │ raise self._make_status_error_from_response(err.response) from None │
│ 1556 │ │ │ │
│ 1557 │ │ │ break │
│ 1558 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
BadRequestError: Error code: 400 - {'error': {'code': 'InvalidParameter', 'param': None, 'message':
'<400> InternalError.Algo.InvalidParameter: Value error, batch size is invalid, it should not be larger
than 10.: input.contents', 'type': 'InvalidParameter'}, 'id': '5aab869a-0505-9449-9a2e-53f6a0f861d0',
'request_id': '5aab869a-0505-9449-9a2e-53f6a0f861d0'}

       During handling of the above exception, another exception occurred:                                     
                                                                                                               
       ╭───────────────────────────────── Traceback (most recent call last) ──────────────────────────────────╮
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/main.py:33 │
       │ 74 in aembedding                                                                                     │
       │                                                                                                      │
       │   3371 │   │   elif isinstance(init_response, EmbeddingResponse):  ## CACHING SCENARIO               │
       │   3372 │   │   │   response = init_response                                                          │
       │   3373 │   │   elif asyncio.iscoroutine(init_response):                                              │
       │ ❱ 3374 │   │   │   response = await init_response  # type: ignore                                    │
       │   3375 │   │   if (                                                                                  │
       │   3376 │   │   │   response is not None                                                              │
       │   3377 │   │   │   and isinstance(response, EmbeddingResponse)                                       │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
       │ i/openai.py:1161 in aembedding                                                                       │
       │                                                                                                      │
       │   1158 │   │   │   error_response = getattr(e, "response", None)                                     │
       │   1159 │   │   │   if error_headers is None and error_response:                                      │
       │   1160 │   │   │   │   error_headers = getattr(error_response, "headers", None)                      │
       │ ❱ 1161 │   │   │   raise OpenAIError(                                                                │
       │   1162 │   │   │   │   status_code=status_code, message=error_text, headers=error_headers            │
       │   1163 │   │   │   )                                                                                 │
       │   1164                                                                                               │
       ╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
       OpenAIError: Error code: 400 - {'error': {'code': 'InvalidParameter', 'param': None, 'message': '<400>  
       InternalError.Algo.InvalidParameter: Value error, batch size is invalid, it should not be larger than   
       10.: input.contents', 'type': 'InvalidParameter'}, 'id': '5aab869a-0505-9449-9a2e-53f6a0f861d0',        
       'request_id': '5aab869a-0505-9449-9a2e-53f6a0f861d0'}                                                   
                                                                                                               
       During handling of the above exception, another exception occurred:                                     
                                                                                                               
       ╭───────────────────────────────── Traceback (most recent call last) ──────────────────────────────────╮
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/paperqa/agents/sea │
       │ rch.py:528 in process_file                                                                           │
       │                                                                                                      │
       │   525 │   │   │                                                                                      │
       │   526 │   │   │   tmp_docs = Docs()                                                                  │
       │   527 │   │   │   try:                                                                               │
       │ ❱ 528 │   │   │   │   await tmp_docs.aadd(                                                           │
       │   529 │   │   │   │   │   path=abs_file_path,                                                        │
       │   530 │   │   │   │   │   fields=["title", "author", "journal", "year"],                             │
       │   531 │   │   │   │   │   settings=settings,                                                         │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/paperqa/docs.py:42 │
       │ 3 in aadd                                                                                            │
       │                                                                                                      │
       │   420 │   │   │   │   f"This does not look like a text document: {path}. Pass disable_check"         │
       │   421 │   │   │   │   " to ignore this error."                                                       │
       │   422 │   │   │   )                                                                                  │
       │ ❱ 423 │   │   if await self.aadd_texts(texts, doc, all_settings, embedding_model):                   │
       │   424 │   │   │   return docname                                                                     │
       │   425 │   │   return None                                                                            │
       │   426                                                                                                │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/paperqa/docs.py:48 │
       │ 1 in aadd_texts                                                                                      │
       │                                                                                                      │
       │   478 │   │   if embedding_model and texts[0].embedding is None:                                     │
       │   479 │   │   │   for t, t_embedding in zip(                                                         │
       │   480 │   │   │   │   texts,                                                                         │
       │ ❱ 481 │   │   │   │   await embedding_model.embed_documents(texts=[t.text for t in texts]),          │
       │   482 │   │   │   │   strict=True,                                                                   │
       │   483 │   │   │   ):                                                                                 │
       │   484 │   │   │   │   t.embedding = t_embedding                                                      │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/lmi/embeddings.py: │
       │ 166 in embed_documents                                                                               │
       │                                                                                                      │
       │   163 │   │   │   │   )                                                                              │
       │   164 │   │   │   )                                                                                  │
       │   165 │   │   │                                                                                      │
       │ ❱ 166 │   │   │   response = await track_costs(self.router.aembedding)(                              │
       │   167 │   │   │   │   model=self.name,                                                               │
       │   168 │   │   │   │   input=texts[i : i + batch_size],                                               │
       │   169 │   │   │   │   dimensions=self.ndim,                                                          │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/lmi/cost_tracker.p │
       │ y:91 in wrapped_func                                                                                 │
       │                                                                                                      │
       │    88 │   """                                                                                        │
       │    89 │                                                                                              │
       │    90 │   async def wrapped_func(*args, **kwargs):                                                   │
       │ ❱  91 │   │   response = await func(*args, **kwargs)                                                 │
       │    92 │   │   if GLOBAL_COST_TRACKER.enabled.get():                                                  │
       │    93 │   │   │   GLOBAL_COST_TRACKER.record(response)                                               │
       │    94 │   │   return response                                                                        │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 2679 in aembedding                                                                                   │
       │                                                                                                      │
       │   2676 │   │   │   │   │   original_exception=e,                                                     │
       │   2677 │   │   │   │   )                                                                             │
       │   2678 │   │   │   )                                                                                 │
       │ ❱ 2679 │   │   │   raise e                                                                           │
       │   2680 │                                                                                             │
       │   2681 │   async def _aembedding(self, input: Union[str, List], model: str, **kwargs):               │
       │   2682 │   │   model_name = None                                                                     │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 2668 in aembedding                                                                                   │
       │                                                                                                      │
       │   2665 │   │   │   kwargs["input"] = input                                                           │
       │   2666 │   │   │   kwargs["original_function"] = self._aembedding                                    │
       │   2667 │   │   │   self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)                  │
       │ ❱ 2668 │   │   │   response = await self.async_function_with_fallbacks(**kwargs)                     │
       │   2669 │   │   │   return response                                                                   │
       │   2670 │   │   except Exception as e:                                                                │
       │   2671 │   │   │   asyncio.create_task(                                                              │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 3542 in async_function_with_fallbacks                                                                │
       │                                                                                                      │
       │   3539 │   │   │   │   │   │   )                                                                     │
       │   3540 │   │   │   │   │   )                                                                         │
       │   3541 │   │   │                                                                                     │
       │ ❱ 3542 │   │   │   raise original_exception                                                          │
       │   3543 │                                                                                             │
       │   3544 │   def _handle_mock_testing_fallbacks(                                                       │
       │   3545 │   │   self,                                                                                 │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 3356 in async_function_with_fallbacks                                                                │
       │                                                                                                      │
       │   3353 │   │   │   │   │   *args, **kwargs, mock_timeout=mock_timeout                                │
       │   3354 │   │   │   │   )                                                                             │
       │   3355 │   │   │   else:                                                                             │
       │ ❱ 3356 │   │   │   │   response = await self.async_function_with_retries(*args, **kwargs)            │
       │   3357 │   │   │   verbose_router_logger.debug(f"Async Response: {response}")                        │
       │   3358 │   │   │   response = add_fallback_headers_to_response(                                      │
       │   3359 │   │   │   │   response=response,                                                            │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 3734 in async_function_with_retries                                                                  │
       │                                                                                                      │
       │   3731 │   │   │   │   setattr(original_exception, "max_retries", num_retries)                       │
       │   3732 │   │   │   │   setattr(original_exception, "num_retries", current_attempt)                   │
       │   3733 │   │   │                                                                                     │
       │ ❱ 3734 │   │   │   raise original_exception                                                          │
       │   3735 │                                                                                             │
       │   3736 │   async def make_call(self, original_function: Any, *args, **kwargs):                       │
       │   3737 │   │   """                                                                                   │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 3625 in async_function_with_retries                                                                  │
       │                                                                                                      │
       │   3622 │   │   │   │   model_group=model_group, kwargs=kwargs                                        │
       │   3623 │   │   │   )                                                                                 │
       │   3624 │   │   │   # if the function call is successful, no exception will be raised and we'll       │
       │        break out of the loop                                                                         │
       │ ❱ 3625 │   │   │   response = await self.make_call(original_function, *args, **kwargs)               │
       │   3626 │   │   │   response = add_retry_headers_to_response(                                         │
       │   3627 │   │   │   │   response=response, attempted_retries=0, max_retries=None                      │
       │   3628 │   │   │   )                                                                                 │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 3743 in make_call                                                                                    │
       │                                                                                                      │
       │   3740 │   │   model_group = kwargs.get("model")                                                     │
       │   3741 │   │   response = original_function(*args, **kwargs)                                         │
       │   3742 │   │   if inspect.iscoroutinefunction(response) or inspect.isawaitable(response):            │
       │ ❱ 3743 │   │   │   response = await response                                                         │
       │   3744 │   │   ## PROCESS RESPONSE HEADERS                                                           │
       │   3745 │   │   response = await self.set_response_headers(                                           │
       │   3746 │   │   │   response=response, model_group=model_group                                        │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 2749 in _aembedding                                                                                  │
       │                                                                                                      │
       │   2746 │   │   │   )                                                                                 │
       │   2747 │   │   │   if model_name is not None:                                                        │
       │   2748 │   │   │   │   self.fail_calls[model_name] += 1                                              │
       │ ❱ 2749 │   │   │   raise e                                                                           │
       │   2750 │                                                                                             │
       │   2751 │   #### FILES API ####                                                                       │
       │   2752 │   async def acreate_file(                                                                   │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 2736 in _aembedding                                                                                  │
       │                                                                                                      │
       │   2733 │   │   │   │   await self.async_routing_strategy_pre_call_checks(                            │
       │   2734 │   │   │   │   │   deployment=deployment, parent_otel_span=parent_otel_span                  │
       │   2735 │   │   │   │   )                                                                             │
       │ ❱ 2736 │   │   │   │   response = await response                                                     │
       │   2737 │   │   │                                                                                     │
       │   2738 │   │   │   self.success_calls[model_name] += 1                                               │
       │   2739 │   │   │   verbose_router_logger.info(                                                       │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/utils.py:1 │
       │ 494 in wrapper_async                                                                                 │
       │                                                                                                      │
       │   1491 │   │   │                                                                                     │
       │   1492 │   │   │   timeout = _get_wrapper_timeout(kwargs=kwargs, exception=e)                        │
       │   1493 │   │   │   setattr(e, "timeout", timeout)                                                    │
       │ ❱ 1494 │   │   │   raise e                                                                           │
       │   1495 │                                                                                             │
       │   1496 │   is_coroutine = inspect.iscoroutinefunction(original_function)                             │
       │   1497                                                                                               │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/utils.py:1 │
       │ 355 in wrapper_async                                                                                 │
       │                                                                                                      │
       │   1352 │   │   │   │   return _caching_handler_response.final_embedding_cached_response              │
       │   1353 │   │   │                                                                                     │
       │   1354 │   │   │   # MODEL CALL                                                                      │
       │ ❱ 1355 │   │   │   result = await original_function(*args, **kwargs)                                 │
       │   1356 │   │   │   end_time = datetime.datetime.now()                                                │
       │   1357 │   │   │   if "stream" in kwargs and kwargs["stream"] is True:                               │
       │   1358 │   │   │   │   if (                                                                          │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/main.py:33 │
       │ 89 in aembedding                                                                                     │
       │                                                                                                      │
       │   3386 │   │   return response                                                                       │
       │   3387 │   except Exception as e:                                                                    │
       │   3388 │   │   custom_llm_provider = custom_llm_provider or "openai"                                 │
       │ ❱ 3389 │   │   raise exception_type(                                                                 │
       │   3390 │   │   │   model=model,                                                                      │
       │   3391 │   │   │   custom_llm_provider=custom_llm_provider,                                          │
       │   3392 │   │   │   original_exception=e,                                                             │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/litellm_co │
       │ re_utils/exception_mapping_utils.py:2239 in exception_type                                           │
       │                                                                                                      │
       │   2236 │   │   # don't let an error with mapping interrupt the user from receiving an error          │
       │        from the llm api calls                                                                        │
       │   2237 │   │   if exception_mapping_worked:                                                          │
       │   2238 │   │   │   setattr(e, "litellm_response_headers", litellm_response_headers)                  │
       │ ❱ 2239 │   │   │   raise e                                                                           │
       │   2240 │   │   else:                                                                                 │
       │   2241 │   │   │   for error_type in litellm.LITELLM_EXCEPTION_TYPES:                                │
       │   2242 │   │   │   │   if isinstance(e, error_type):                                                 │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/litellm_co │
       │ re_utils/exception_mapping_utils.py:402 in exception_type                                            │
       │                                                                                                      │
       │    399 │   │   │   │   │   exception_mapping_worked = True                                           │
       │    400 │   │   │   │   │   if original_exception.status_code == 400:                                 │
       │    401 │   │   │   │   │   │   exception_mapping_worked = True                                       │
       │ ❱  402 │   │   │   │   │   │   raise BadRequestError(                                                │
       │    403 │   │   │   │   │   │   │   message=f"{exception_provider} - {message}",                      │
       │    404 │   │   │   │   │   │   │   llm_provider=custom_llm_provider,                                 │
       │    405 │   │   │   │   │   │   │   model=model,                                                      │
       ╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
       BadRequestError: litellm.BadRequestError: OpenAIException - Error code: 400 - {'error': {'code':        
       'InvalidParameter', 'param': None, 'message': '<400> InternalError.Algo.InvalidParameter: Value error,  
       batch size is invalid, it should not be larger than 10.: input.contents', 'type': 'InvalidParameter'},  
       'id': '5aab869a-0505-9449-9a2e-53f6a0f861d0', 'request_id': '5aab869a-0505-9449-9a2e-53f6a0f861d0'}.    
       Received Model Group=openai/text-embedding-v4                                                           
       Available Model Group Fallbacks=None LiteLLM Retried: 2 times, LiteLLM Max Retries: 3

[jamesbraza]

Hi @zigerZZZ ah yeah our default batch size was 16. Future-House/ldp#301 makes it configurable. Until it's merged/released, to get this:

1. pip install the dev version

pip install git+https://github.com/Future-House/ldp.git@fixing-batch-size#subdirectory=packages/lmi

2. Configure embedding_config setting to include a batch size of 8: https://github.com/Future-House/paper-qa/tree/v5.21.0?tab=readme-ov-file#settings-cheatsheet

[zigerZZZ]

if embedding_model and texts[0].embedding is None: batch_size = 10 # Set batch size to avoid API errors text_embeddings = [] for i in range(0, len(texts), batch_size): batch = texts[i : i + batch_size] batch_embeddings = await embedding_model.embed_documents( texts=[t.text for t in batch] ) text_embeddings.extend(batch_embeddings) for t, t_embedding in zip( texts, text_embeddings, strict=True, ): t.embedding = t_embedding

replace paperqa\docs.py

thx! but another problem occurs, it's BadRequestError saying that batch size is invalid, it should not be larger than 10谢谢！但是发生了另一个问题，这是BadrequeStror，说批处理大小无效，它不应大于10 Shall I change to any other LLM?我可以更改为其他LLM吗？
[09:55:22] Error parsing PaperQA2.pdf, skipping index for this file.
╭───────────────────────────────── Traceback (most recent call last) ──────────────────────────────────╮
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
│ i/openai.py:1116 in aembedding │
│ │
│ 1113 │ │ │ │ max_retries=max_retries, │
│ 1114 │ │ │ │ client=client, │
│ 1115 │ │ │ ) │
│ ❱ 1116 │ │ │ headers, response = await self.make_openai_embedding_request( │
│ 1117 │ │ │ │ openai_aclient=openai_aclient, │
│ 1118 │ │ │ │ data=data, │
│ 1119 │ │ │ │ timeout=timeout, │
│ │
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/litellm_co │
│ re_utils/logging_utils.py:135 in async_wrapper │
│ │
│ 132 │ │ async def async_wrapper(*args, **kwargs): │
│ 133 │ │ │ start_time = datetime.now() │
│ 134 │ │ │ try: │
│ ❱ 135 │ │ │ │ result = await func(*args, **kwargs) │
│ 136 │ │ │ │ return result │
│ 137 │ │ │ finally: │
│ 138 │ │ │ │ end_time = datetime.now() │
│ │
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
│ i/openai.py:1069 in make_openai_embedding_request │
│ │
│ 1066 │ │ │ response = raw_response.parse() │
│ 1067 │ │ │ return headers, response │
│ 1068 │ │ except Exception as e: │
│ ❱ 1069 │ │ │ raise e │
│ 1070 │ │
│ 1071 │ @track_llm_api_timing() │
│ 1072 │ def make_sync_openai_embedding_request( │
│ │
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
│ i/openai.py:1062 in make_openai_embedding_request │
│ │
│ 1059 │ │ - call embeddings.create by default │
│ 1060 │ │ """ │
│ 1061 │ │ try: │
│ ❱ 1062 │ │ │ raw_response = await openai_aclient.embeddings.with_raw_response.create( │
│ 1063 │ │ │ │ **data, timeout=timeout │
│ 1064 │ │ │ ) # type: ignore │
│ 1065 │ │ │ headers = dict(raw_response.headers) │
│ │
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/_legacy_res │
│ ponse.py:381 in wrapped │
│ │
│ 378 │ │ │
│ 379 │ │ kwargs["extra_headers"] = extra_headers │
│ 380 │ │ │
│ ❱ 381 │ │ return cast(LegacyAPIResponse[R], await func(*args, **kwargs)) │
│ 382 │ │
│ 383 │ return wrapped │
│ 384 │
│ │
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/resources/e │
│ mbeddings.py:245 in create │
│ │
│ 242 │ │ │ │
│ 243 │ │ │ return obj │
│ 244 │ │ │
│ ❱ 245 │ │ return await self._post( │
│ 246 │ │ │ "/embeddings", │
│ 247 │ │ │ body=maybe_transform(params, embedding_create_params.EmbeddingCreateParams), │
│ 248 │ │ │ options=make_request_options( │
│ │
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/_base_clien │
│ t.py:1748 in post │
│ │
│ 1745 │ │ opts = FinalRequestOptions.construct( │
│ 1746 │ │ │ method="post", url=path, json_data=body, files=await │
│ async_to_httpx_files(files), **options │
│ 1747 │ │ ) │
│ ❱ 1748 │ │ return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls) │
│ 1749 │ │
│ 1750 │ async def patch( │
│ 1751 │ │ self, │
│ │
│ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/openai/_base_clien │
│ t.py:1555 in request │
│ │
│ 1552 │ │ │ │ │ await err.response.aread() │
│ 1553 │ │ │ │ │
│ 1554 │ │ │ │ log.debug("Re-raising status error") │
│ ❱ 1555 │ │ │ │ raise self._make_status_error_from_response(err.response) from None │
│ 1556 │ │ │ │
│ 1557 │ │ │ break │
│ 1558 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
BadRequestError: Error code: 400 - {'error': {'code': 'InvalidParameter', 'param': None, 'message':
'<400> InternalError.Algo.InvalidParameter: Value error, batch size is invalid, it should not be larger
than 10.: input.contents', 'type': 'InvalidParameter'}, 'id': '5aab869a-0505-9449-9a2e-53f6a0f861d0',
'request_id': '5aab869a-0505-9449-9a2e-53f6a0f861d0'}

       During handling of the above exception, another exception occurred:                                     
                                                                                                               
       ╭───────────────────────────────── Traceback (most recent call last) ──────────────────────────────────╮
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/main.py:33 │
       │ 74 in aembedding                                                                                     │
       │                                                                                                      │
       │   3371 │   │   elif isinstance(init_response, EmbeddingResponse):  ## CACHING SCENARIO               │
       │   3372 │   │   │   response = init_response                                                          │
       │   3373 │   │   elif asyncio.iscoroutine(init_response):                                              │
       │ ❱ 3374 │   │   │   response = await init_response  # type: ignore                                    │
       │   3375 │   │   if (                                                                                  │
       │   3376 │   │   │   response is not None                                                              │
       │   3377 │   │   │   and isinstance(response, EmbeddingResponse)                                       │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/llms/opena │
       │ i/openai.py:1161 in aembedding                                                                       │
       │                                                                                                      │
       │   1158 │   │   │   error_response = getattr(e, "response", None)                                     │
       │   1159 │   │   │   if error_headers is None and error_response:                                      │
       │   1160 │   │   │   │   error_headers = getattr(error_response, "headers", None)                      │
       │ ❱ 1161 │   │   │   raise OpenAIError(                                                                │
       │   1162 │   │   │   │   status_code=status_code, message=error_text, headers=error_headers            │
       │   1163 │   │   │   )                                                                                 │
       │   1164                                                                                               │
       ╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
       OpenAIError: Error code: 400 - {'error': {'code': 'InvalidParameter', 'param': None, 'message': '<400>  
       InternalError.Algo.InvalidParameter: Value error, batch size is invalid, it should not be larger than   
       10.: input.contents', 'type': 'InvalidParameter'}, 'id': '5aab869a-0505-9449-9a2e-53f6a0f861d0',        
       'request_id': '5aab869a-0505-9449-9a2e-53f6a0f861d0'}                                                   
                                                                                                               
       During handling of the above exception, another exception occurred:                                     
                                                                                                               
       ╭───────────────────────────────── Traceback (most recent call last) ──────────────────────────────────╮
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/paperqa/agents/sea │
       │ rch.py:528 in process_file                                                                           │
       │                                                                                                      │
       │   525 │   │   │                                                                                      │
       │   526 │   │   │   tmp_docs = Docs()                                                                  │
       │   527 │   │   │   try:                                                                               │
       │ ❱ 528 │   │   │   │   await tmp_docs.aadd(                                                           │
       │   529 │   │   │   │   │   path=abs_file_path,                                                        │
       │   530 │   │   │   │   │   fields=["title", "author", "journal", "year"],                             │
       │   531 │   │   │   │   │   settings=settings,                                                         │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/paperqa/docs.py:42 │
       │ 3 in aadd                                                                                            │
       │                                                                                                      │
       │   420 │   │   │   │   f"This does not look like a text document: {path}. Pass disable_check"         │
       │   421 │   │   │   │   " to ignore this error."                                                       │
       │   422 │   │   │   )                                                                                  │
       │ ❱ 423 │   │   if await self.aadd_texts(texts, doc, all_settings, embedding_model):                   │
       │   424 │   │   │   return docname                                                                     │
       │   425 │   │   return None                                                                            │
       │   426                                                                                                │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/paperqa/docs.py:48 │
       │ 1 in aadd_texts                                                                                      │
       │                                                                                                      │
       │   478 │   │   if embedding_model and texts[0].embedding is None:                                     │
       │   479 │   │   │   for t, t_embedding in zip(                                                         │
       │   480 │   │   │   │   texts,                                                                         │
       │ ❱ 481 │   │   │   │   await embedding_model.embed_documents(texts=[t.text for t in texts]),          │
       │   482 │   │   │   │   strict=True,                                                                   │
       │   483 │   │   │   ):                                                                                 │
       │   484 │   │   │   │   t.embedding = t_embedding                                                      │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/lmi/embeddings.py: │
       │ 166 in embed_documents                                                                               │
       │                                                                                                      │
       │   163 │   │   │   │   )                                                                              │
       │   164 │   │   │   )                                                                                  │
       │   165 │   │   │                                                                                      │
       │ ❱ 166 │   │   │   response = await track_costs(self.router.aembedding)(                              │
       │   167 │   │   │   │   model=self.name,                                                               │
       │   168 │   │   │   │   input=texts[i : i + batch_size],                                               │
       │   169 │   │   │   │   dimensions=self.ndim,                                                          │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/lmi/cost_tracker.p │
       │ y:91 in wrapped_func                                                                                 │
       │                                                                                                      │
       │    88 │   """                                                                                        │
       │    89 │                                                                                              │
       │    90 │   async def wrapped_func(*args, **kwargs):                                                   │
       │ ❱  91 │   │   response = await func(*args, **kwargs)                                                 │
       │    92 │   │   if GLOBAL_COST_TRACKER.enabled.get():                                                  │
       │    93 │   │   │   GLOBAL_COST_TRACKER.record(response)                                               │
       │    94 │   │   return response                                                                        │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 2679 in aembedding                                                                                   │
       │                                                                                                      │
       │   2676 │   │   │   │   │   original_exception=e,                                                     │
       │   2677 │   │   │   │   )                                                                             │
       │   2678 │   │   │   )                                                                                 │
       │ ❱ 2679 │   │   │   raise e                                                                           │
       │   2680 │                                                                                             │
       │   2681 │   async def _aembedding(self, input: Union[str, List], model: str, **kwargs):               │
       │   2682 │   │   model_name = None                                                                     │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 2668 in aembedding                                                                                   │
       │                                                                                                      │
       │   2665 │   │   │   kwargs["input"] = input                                                           │
       │   2666 │   │   │   kwargs["original_function"] = self._aembedding                                    │
       │   2667 │   │   │   self._update_kwargs_before_fallbacks(model=model, kwargs=kwargs)                  │
       │ ❱ 2668 │   │   │   response = await self.async_function_with_fallbacks(**kwargs)                     │
       │   2669 │   │   │   return response                                                                   │
       │   2670 │   │   except Exception as e:                                                                │
       │   2671 │   │   │   asyncio.create_task(                                                              │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 3542 in async_function_with_fallbacks                                                                │
       │                                                                                                      │
       │   3539 │   │   │   │   │   │   )                                                                     │
       │   3540 │   │   │   │   │   )                                                                         │
       │   3541 │   │   │                                                                                     │
       │ ❱ 3542 │   │   │   raise original_exception                                                          │
       │   3543 │                                                                                             │
       │   3544 │   def _handle_mock_testing_fallbacks(                                                       │
       │   3545 │   │   self,                                                                                 │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 3356 in async_function_with_fallbacks                                                                │
       │                                                                                                      │
       │   3353 │   │   │   │   │   *args, **kwargs, mock_timeout=mock_timeout                                │
       │   3354 │   │   │   │   )                                                                             │
       │   3355 │   │   │   else:                                                                             │
       │ ❱ 3356 │   │   │   │   response = await self.async_function_with_retries(*args, **kwargs)            │
       │   3357 │   │   │   verbose_router_logger.debug(f"Async Response: {response}")                        │
       │   3358 │   │   │   response = add_fallback_headers_to_response(                                      │
       │   3359 │   │   │   │   response=response,                                                            │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 3734 in async_function_with_retries                                                                  │
       │                                                                                                      │
       │   3731 │   │   │   │   setattr(original_exception, "max_retries", num_retries)                       │
       │   3732 │   │   │   │   setattr(original_exception, "num_retries", current_attempt)                   │
       │   3733 │   │   │                                                                                     │
       │ ❱ 3734 │   │   │   raise original_exception                                                          │
       │   3735 │                                                                                             │
       │   3736 │   async def make_call(self, original_function: Any, *args, **kwargs):                       │
       │   3737 │   │   """                                                                                   │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 3625 in async_function_with_retries                                                                  │
       │                                                                                                      │
       │   3622 │   │   │   │   model_group=model_group, kwargs=kwargs                                        │
       │   3623 │   │   │   )                                                                                 │
       │   3624 │   │   │   # if the function call is successful, no exception will be raised and we'll       │
       │        break out of the loop                                                                         │
       │ ❱ 3625 │   │   │   response = await self.make_call(original_function, *args, **kwargs)               │
       │   3626 │   │   │   response = add_retry_headers_to_response(                                         │
       │   3627 │   │   │   │   response=response, attempted_retries=0, max_retries=None                      │
       │   3628 │   │   │   )                                                                                 │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 3743 in make_call                                                                                    │
       │                                                                                                      │
       │   3740 │   │   model_group = kwargs.get("model")                                                     │
       │   3741 │   │   response = original_function(*args, **kwargs)                                         │
       │   3742 │   │   if inspect.iscoroutinefunction(response) or inspect.isawaitable(response):            │
       │ ❱ 3743 │   │   │   response = await response                                                         │
       │   3744 │   │   ## PROCESS RESPONSE HEADERS                                                           │
       │   3745 │   │   response = await self.set_response_headers(                                           │
       │   3746 │   │   │   response=response, model_group=model_group                                        │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 2749 in _aembedding                                                                                  │
       │                                                                                                      │
       │   2746 │   │   │   )                                                                                 │
       │   2747 │   │   │   if model_name is not None:                                                        │
       │   2748 │   │   │   │   self.fail_calls[model_name] += 1                                              │
       │ ❱ 2749 │   │   │   raise e                                                                           │
       │   2750 │                                                                                             │
       │   2751 │   #### FILES API ####                                                                       │
       │   2752 │   async def acreate_file(                                                                   │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/router.py: │
       │ 2736 in _aembedding                                                                                  │
       │                                                                                                      │
       │   2733 │   │   │   │   await self.async_routing_strategy_pre_call_checks(                            │
       │   2734 │   │   │   │   │   deployment=deployment, parent_otel_span=parent_otel_span                  │
       │   2735 │   │   │   │   )                                                                             │
       │ ❱ 2736 │   │   │   │   response = await response                                                     │
       │   2737 │   │   │                                                                                     │
       │   2738 │   │   │   self.success_calls[model_name] += 1                                               │
       │   2739 │   │   │   verbose_router_logger.info(                                                       │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/utils.py:1 │
       │ 494 in wrapper_async                                                                                 │
       │                                                                                                      │
       │   1491 │   │   │                                                                                     │
       │   1492 │   │   │   timeout = _get_wrapper_timeout(kwargs=kwargs, exception=e)                        │
       │   1493 │   │   │   setattr(e, "timeout", timeout)                                                    │
       │ ❱ 1494 │   │   │   raise e                                                                           │
       │   1495 │                                                                                             │
       │   1496 │   is_coroutine = inspect.iscoroutinefunction(original_function)                             │
       │   1497                                                                                               │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/utils.py:1 │
       │ 355 in wrapper_async                                                                                 │
       │                                                                                                      │
       │   1352 │   │   │   │   return _caching_handler_response.final_embedding_cached_response              │
       │   1353 │   │   │                                                                                     │
       │   1354 │   │   │   # MODEL CALL                                                                      │
       │ ❱ 1355 │   │   │   result = await original_function(*args, **kwargs)                                 │
       │   1356 │   │   │   end_time = datetime.datetime.now()                                                │
       │   1357 │   │   │   if "stream" in kwargs and kwargs["stream"] is True:                               │
       │   1358 │   │   │   │   if (                                                                          │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/main.py:33 │
       │ 89 in aembedding                                                                                     │
       │                                                                                                      │
       │   3386 │   │   return response                                                                       │
       │   3387 │   except Exception as e:                                                                    │
       │   3388 │   │   custom_llm_provider = custom_llm_provider or "openai"                                 │
       │ ❱ 3389 │   │   raise exception_type(                                                                 │
       │   3390 │   │   │   model=model,                                                                      │
       │   3391 │   │   │   custom_llm_provider=custom_llm_provider,                                          │
       │   3392 │   │   │   original_exception=e,                                                             │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/litellm_co │
       │ re_utils/exception_mapping_utils.py:2239 in exception_type                                           │
       │                                                                                                      │
       │   2236 │   │   # don't let an error with mapping interrupt the user from receiving an error          │
       │        from the llm api calls                                                                        │
       │   2237 │   │   if exception_mapping_worked:                                                          │
       │   2238 │   │   │   setattr(e, "litellm_response_headers", litellm_response_headers)                  │
       │ ❱ 2239 │   │   │   raise e                                                                           │
       │   2240 │   │   else:                                                                                 │
       │   2241 │   │   │   for error_type in litellm.LITELLM_EXCEPTION_TYPES:                                │
       │   2242 │   │   │   │   if isinstance(e, error_type):                                                 │
       │                                                                                                      │
       │ /share/CADD/biotools/anaconda3-2023.09/envs/paper-qa/lib/python3.11/site-packages/litellm/litellm_co │
       │ re_utils/exception_mapping_utils.py:402 in exception_type                                            │
       │                                                                                                      │
       │    399 │   │   │   │   │   exception_mapping_worked = True                                           │
       │    400 │   │   │   │   │   if original_exception.status_code == 400:                                 │
       │    401 │   │   │   │   │   │   exception_mapping_worked = True                                       │
       │ ❱  402 │   │   │   │   │   │   raise BadRequestError(                                                │
       │    403 │   │   │   │   │   │   │   message=f"{exception_provider} - {message}",                      │
       │    404 │   │   │   │   │   │   │   llm_provider=custom_llm_provider,                                 │
       │    405 │   │   │   │   │   │   │   model=model,                                                      │
       ╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
       BadRequestError: litellm.BadRequestError: OpenAIException - Error code: 400 - {'error': {'code':        
       'InvalidParameter', 'param': None, 'message': '<400> InternalError.Algo.InvalidParameter: Value error,  
       batch size is invalid, it should not be larger than 10.: input.contents', 'type': 'InvalidParameter'},  
       'id': '5aab869a-0505-9449-9a2e-53f6a0f861d0', 'request_id': '5aab869a-0505-9449-9a2e-53f6a0f861d0'}.    
       Received Model Group=openai/text-embedding-v4                                                           
       Available Model Group Fallbacks=None LiteLLM Retried: 2 times, LiteLLM Max Retries: 3

it works

[zigerZZZ]

Hi @zigerZZZ ah yeah our default batch size was 16. Future-House/ldp#301 makes it configurable. Until it's merged/released, to get this:

1. pip install the dev version

pip install git+https://github.com/Future-House/ldp.git@fixing-batch-size#subdirectory=packages/lmi
2. Configure embedding_config setting to include a batch size of 8: https://github.com/Future-House/paper-qa/tree/v5.21.0?tab=readme-ov-file#settings-cheatsheet

sure, i will close this, if it has been merged.Thanks a lot

[jamesbraza]

Alright you can now do pip install fhlmi>=0.29.0 and batch size is configurable within PaperQA's embedding_config setting.

I am going to close this out, please feel free to open a new issue if you keep hitting these failures