Connection to LLm in Version 2.x — Works in 1.7, Fails in Latest

[mycybercatnet]

--

Subject: API Connection Issue in Version 2.x — Works in 1.7, Fails in Latest

Description:

In the latest version of the application (>=2.0), I’m unable to connect to either Ollama or OpenAI backends. Everything works as expected in version 1.7, with identical configuration and environment.

---

Environment Variables:

OPENAI_ENDPOINT=http://192.168.1.54:8163/v1
OPENAI_API_KEY=************************************

OLLAMA_ENDPOINT=http://192.168.1.80:11434

ANTHROPIC_ENDPOINT=https://api.anthropic.com
ANTHROPIC_API_KEY=

GOOGLE_API_KEY=

AZURE_OPENAI_ENDPOINT=
AZURE_OPENAI_API_KEY=
AZURE_OPENAI_API_VERSION=2025-01-01-preview

DEEPSEEK_ENDPOINT=https://api.deepseek.com
DEEPSEEK_API_KEY=

MISTRAL_API_KEY=
MISTRAL_ENDPOINT=https://api.mistral.ai/v1

---

Observed Behavior:

• In version 2.x, the app fails to connect to both OLLAMA_ENDPOINT and OPENAI_ENDPOINT.
• No output or error logs are clearly visible — just silent failure or fallback.
• Rolling back to version 1.7 restores full functionality with the same environment and API endpoints.

---

Expected Behavior:

• App should establish connections to Ollama and OpenAI backends using the provided endpoints.
• At minimum, meaningful error logs should appear if the connection fails.

---

Additional Notes:

• Network, API keys, and server accessibility have been verified.
• Equipment, configuration, and environment variables are identical between version 1.7 and 2.x.

---

Could you please investigate what changed in the connection logic or model provider handling in version 2.x?

---

Хочешь, я добавлю скриншот или пример curl-запроса к API для доказательства, что endpoint доступен?

[mycybercatnet]

INFO [agent] 🚀 Starting task: open google
INFO [agent] 📍 Step 1
ERROR [agent] ❌ Result failed 1/3 times:
Error code: 422 - {'detail': [{'loc': ['body', 'messages', 3, 'content', 'str'], 'message': 'Input should be a valid string', 'type': 'string_type'}, {'loc': ['body', 'messages', 3, 'content', 'list[dict[str,union[str,dict[str,str]]]]'], 'message': 'Input should be a valid list', 'type': 'list_type'}, {'loc': ['body', 'tool_choice'], 'message': 'Input should be a valid string', 'type': 'string_type'}]}
INFO [agent] 📍 Step 1
ERROR [agent] ❌ Result failed 2/3 times:
Error code: 422 - {'detail': [{'loc': ['body', 'messages', 3, 'content', 'str'], 'message': 'Input should be a valid string', 'type': 'string_type'}, {'loc': ['body', 'messages', 3, 'content', 'list[dict[str,union[str,dict[str,str]]]]'], 'message': 'Input should be a valid list', 'type': 'list_type'}, {'loc': ['body', 'tool_choice'], 'message': 'Input should be a valid string', 'type': 'string_type'}]}
INFO [agent] 📍 Step 1
ERROR [agent] ❌ Result failed 3/3 times:
Error code: 422 - {'detail': [{'loc': ['body', 'messages', 3, 'content', 'str'], 'message': 'Input should be a valid string', 'type': 'string_type'}, {'loc': ['body', 'messages', 3, 'content', 'list[dict[str,union[str,dict[str,str]]]]'], 'message': 'Input should be a valid list', 'type': 'list_type'}, {'loc': ['body', 'tool_choice'], 'message': 'Input should be a valid string', 'type': 'string_type'}]}
ERROR [src.agent.browser_use.browser_use_agent] ❌ Stopping due to 3 consecutive failures
INFO [agent] Created GIF at ./tmp/agent_history\9c50b2ee-d3e3-4d7f-a820-46f89f4d37cf\9c50b2ee-d3e3-4d7f-a820-46f89f4d37cf.gif
INFO [src.webui.components.browser_use_agent_tab] Agent task completing...

[mycybercatnet]

Hi team,
I've been working with a custom fork of browser-use and encountered consistent issues where the agent fails on the first step of any task.

🛠 What I did
✅ Setup and Agent Initialization Works Fine:
I correctly launch the UI via webui.py.

Chrome launches (non-headless), browser context is created, and agent starts.

The interface displays all UI components and responds to button events.

✅ LLM Works (with wrapper):
I'm using a custom LLM-compatible API (OpenAI-style), and I created a wrapper called SafeLLMWrapper to sanitize incoming messages before sending them to invoke().

Here’s the implementation:

python
Копировать
Редактировать
from langchain_core.language_models.chat_models import BaseChatModel
from langchain_core.messages import BaseMessage
from pydantic import PrivateAttr
from typing import Any, List

class SafeLLMWrapper(BaseChatModel):
_base_llm: BaseChatModel = PrivateAttr()

def __init__(self, base_llm: BaseChatModel):
    super().__init__()
    self._base_llm = base_llm

@property
def _llm_type(self) -> str:
    return "safe-wrapper"

def _clean_messages(self, messages: List[BaseMessage]) -> List[BaseMessage]:
    for msg in messages:
        if msg.content is None:
            msg.content = ""
        elif not isinstance(msg.content, (str, list)):
            msg.content = str(msg.content)

        if isinstance(msg.content, list):
            cleaned = []
            for item in msg.content:
                if isinstance(item, dict) and "type" in item and (
                    "text" in item or "image_url" in item
                ):
                    cleaned.append(item)
            msg.content = cleaned or "🛑 invalid multimodal content removed"
    return messages

def invoke(self, messages: List[BaseMessage], *args: Any, **kwargs: Any) -> Any:
    clean = self._clean_messages(messages)
    print("🟨 [LLM] invoke → Cleaned messages:", clean)
    result = self._base_llm.invoke(clean, *args, **kwargs)
    print("🟩 [LLM] invoke → Result:", result)
    return result

def _generate(self, messages: List[BaseMessage], **kwargs: Any) -> Any:
    messages = self._clean_messages(messages)
    return self._base_llm._generate(messages, **kwargs)

async def _agenerate(self, messages: List[BaseMessage], **kwargs: Any) -> Any:
    messages = self._clean_messages(messages)
    return await self._base_llm._agenerate(messages, **kwargs)

This wrapper ensures that msg.content is always a string or a valid multimodal list, stripping anything invalid. It logs both cleaned inputs and outputs to stdout.

🧪 The Result
LLM responses are valid and returned correctly (e.g., "Paris" in a simple test prompt).

But: no matter what prompt I give the agent ("open google.com", "перейди на сайт гугл", etc.), it fails the first step 3 times in a row and stops:

vbnet
Копировать
Редактировать
INFO Starting task: открой гугл
INFO Step 1
ERROR ❌ Result failed 1/3 times
ERROR ❌ Result failed 2/3 times
ERROR ❌ Result failed 3/3 times
ERROR Stopping due to 3 consecutive failures
A .gif is saved, a JSON is generated — but no useful action is executed.

No error traceback from the LLM — it appears to respond fine.

❓What could be wrong?
Is there a system prompt mismatch? I tried overriding/extending it — no success.

Is the agent expecting a very specific reply schema that GPT-4o doesn’t satisfy?

Do I need to adjust tool_calling_method or step planner settings manually?

I’m happy to test and debug further — but I feel like the planner or tool execution pipeline silently fails somewhere.

Thanks for the amazing tool. Any pointers are appreciated!