Skip to content

WebSurferAgent#

Open In Colab Open on GitHub

In browser-use tool and crawl4ai tool notebooks, we demonstrated how to create Agents with basic web surfing capabilities.

Now, we’re taking it a step further with WebSurferAgent—a powerful agent equipped with built-in web surfing tools right out of the box!

WebSurferAgent with browser-use tool#

Warning: Browser Use requires Python 3.11 or higher.

Installation#

To get started with the browser-use integration in AG2, follow these steps:

  1. Install AG2 with the browser-use extra:

    pip install -U ag2[openai,browser-use]

    Note: If you have been using autogen or pyautogen, all you need to do is upgrade it using:

    pip install -U autogen[openai,browser-use]

    or

    pip install -U pyautogen[openai,browser-use]

    as pyautogen, autogen, and ag2 are aliases for the same PyPI package.

  2. Set up Playwright:

    # Installs Playwright and browsers for all OS
playwright install
# Additional command, mandatory for Linux only
playwright install-deps

  3. For running the code in Jupyter, use nest_asyncio to allow nested event loops. bash pip install nest_asyncio

You’re all set! Now you can start using browsing features in AG2.

Imports#

import os

import nest_asyncio

from autogen.agentchat import UserProxyAgent
from autogen.agents.experimental import WebSurferAgent

nest_asyncio.apply()

browser-use WebSurferAgent#

Note: Browser Use supports the following models: Supported Models

We had great experience with OpenAI, Anthropic, and Gemini. However, DeepSeek and Ollama haven’t performed as well.

config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]

llm_config = {
    "config_list": config_list,
}

There are two ways to start a chat session which is using only one agent with LLM configuration.

The new run method simplifies the process by eliminating the need for manual UserProxyAgent creation.

  • ✅ Easier setup – No need to manually register tools
# The `web_tool="browser_use"` tells the agent to use the `BrowserUseTool` to surf the web.
websurfer = WebSurferAgent(name="WebSurfer", llm_config=llm_config, web_tool="browser_use")

run_response = websurfer.run(
    message="Get info from https://docs.ag2.ai/docs/Home",
    tools=websurfer.tools,
    max_turns=2,
    user_input=False,
)
run_response.process()

Manual Setup: Using initiate_chat Method#

This method requires manually creating a UserProxyAgent and registering tools for execution.

  • ⚠️ More setup required
  • ⚠️ Must manually register tools
websurfer = WebSurferAgent(name="WebSurfer", llm_config=llm_config, web_tool="browser_use")
user_proxy = UserProxyAgent(name="user_proxy", human_input_mode="NEVER")
# WebSurferAgent has a list of tools which are registered for LLM
# We need to register the tools for execution with the UserProxyAgent
for tool in websurfer.tools:
    tool.register_for_execution(user_proxy)

user_proxy.initiate_chat(
    recipient=websurfer,
    message="Get info from https://docs.ag2.ai/docs/Home",
    max_turns=2,
)

WebSurferAgent with crawl4ai tool#

Installation#

To get started with the crawl4ai integration in AG2, follow these steps:

  1. Install AG2 with the crawl4ai extra:

    pip install -U ag2[openai,crawl4ai]

    Note: If you have been using autogen or pyautogen, all you need to do is upgrade it using:

    pip install -U autogen[openai,crawl4ai]

    or

    pip install -U pyautogen[openai,crawl4ai]

    as pyautogen, autogen, and ag2 are aliases for the same PyPI package.

  2. Set up Playwright:

    # Installs Playwright and browsers for all OS
playwright install
# Additional command, mandatory for Linux only
playwright install-deps

  3. For running the code in Jupyter, use nest_asyncio to allow nested event loops. bash pip install nest_asyncio

You’re all set! Now you can start using browsing features in AG2.

Imports#

import os

import nest_asyncio

from autogen.agentchat import UserProxyAgent
from autogen.agents.experimental import WebSurferAgent

nest_asyncio.apply()

Crawl4AI WebSurferAgent#

Note: Crawl4AI is built on top of LiteLLM and supports the same models as LiteLLM.

We had great experience with OpenAI, Anthropic, Gemini and Ollama. However, as of this writing, DeepSeek is encountering some issues.

config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]

llm_config = {
    "config_list": config_list,
}

There are two ways to start a chat session which is using only one agent with LLM configuration.

The new run method simplifies the process by eliminating the need for manual UserProxyAgent creation.

  • ✅ Easier setup – No need to manually register tools
# `web_tool` parameter must be set to `crawl4ai` in order for the `Crawl4AITool` to be used.
websurfer = WebSurferAgent(name="WebSurfer", llm_config=llm_config, web_tool="crawl4ai")

run_response = websurfer.run(
    message="Get info from https://docs.ag2.ai/docs/Home",
    tools=websurfer.tools,
    max_turns=2,
    user_input=False,
)
run_response.process()

Manual Setup: Using initiate_chat Method#

This method requires manually creating a UserProxyAgent and registering tools for execution.

  • ⚠️ More setup required
  • ⚠️ Must manually register tools
user_proxy = UserProxyAgent(name="user_proxy", human_input_mode="NEVER")
websurfer = WebSurferAgent(name="WebSurfer", llm_config=llm_config, web_tool="crawl4ai")

# WebSurferAgent has a list of tools which are registered for LLM
# We need to register the tools for execution with the UserProxyAgent
for tool in websurfer.tools:
    tool.register_for_execution(user_proxy)

user_proxy.initiate_chat(
    recipient=websurfer,
    message="Get info from https://docs.ag2.ai/docs/Home",
    max_turns=2,
)