Develop a version of WebSurferAgent that uses a proper headless browser (perhaps with multimodal LLM support)

[afourney]

Possible roadmap:

• Investigate (Headless Chrome)[https://developer.chrome.com/blog/headless-chrome/] or equivalent.
• Add a HeadlessChromeBrowser to complement SimpleTextBrowser in https://github.com/microsoft/autogen/blob/main/autogen/browser_utils.py
• Update WebSurferAgent to accept a web_broswer instance rather than a web_browser_config, and pass in either a SimpleTextBrowser or HeadlessChromeBrowser as appropriate

Additional thoughts: We should try to take full advantage to having a browser under our control. Don't just dump the dom to HTML for BeautifulSoup to parse (like what Langchain does). Rather use javascript running privileged in the page context to query the document, extract text, interact with links, etc.

[vijaykramesh]

Started an initial pass at this, right now it doesn't do much more than the SimpleTextBrowser but since it runs in a headless chrome it can take advantage of client side interactions. It might also be worth looking into something like https://github.com/lucgagan/auto-playwright to allow the HeadlessChromeBrowser agent to intelligently do selenium interactions.

[gee842]

I have opened this issue which might be related as it tackles a similar problem: #1538

However, one enhancement that would be extremely beneficial would be shortening the context length by selectively remembering things and increasing the useful-information-density produced by the web browsing agent.

Perhaps a library like this might come in handy:

https://github.com/buriy/python-readability

[afourney]

I have opened this issue which might be related as it tackles a similar problem: #1538

However, one enhancement that would be extremely beneficial would be shortening the context length by selectively remembering things and increasing the useful-information-density produced by the web browsing agent.

Perhaps a library like this might come in handy:

https://github.com/buriy/python-readability

Yes, 100%. If our goal is to find a piece of information on a webpage, we don't need to keep the whole browsing history in our context window. If one page/search result was a dead-end, it's just burning tokens to keep that around when surfing to other pages.

Gagan's PR might help (if we limit the context to a few messages #1513 ) But additional strategies like summarization would no doubt also be advantageous.

FWIW the reason SimpleTextBrowser provides virtual "viewports" is an effort to help with this issue (by not immediately swamping the context with long pages)