WebSurferAgent

If you need an agent that can browse, extract, or interact with the web, WebSurferAgent is a good choice. The agent actions the request(s) given to it by determining what to do on the web and browsing and crawling it, returning the details of what it finds.

The WebSurferAgent has two in-built web tools to choose from:

1. browser-use - uses an actual browser instance (visible or headless), interacting with the web pages in realtime
2. Crawl4AI - crawls without a visual browser instance

To get started with WebSurferAgent, install AG2 with the browser-use and/or crawl4ai extras.

pip install ag2[browser-use]

and/or

pip install ag2[crawl4ai]

And then setup Playwright:

playwright install

Now, you can create an agent, nominating the web tool:

from autogen.agents import WebSurferAgent

llm_config = {"model": "gpt-4o-mini"}

# Create our agent
websurfer = WebSurferAgent(
    name="WebSurfer",
    llm_config=llm_config,
    web_tool="browser_use",
)
# or
websurfer = WebSurferAgent(
    name="WebSurfer",
    llm_config=llm_config,
    web_tool="crawl4ai",
)

Let’s browse the web for news on AG2.

# 1. Import our WebSurferAgent
from autogen.agents import WebSurferAgent

llm_config = {"model": "gpt-4o-mini"}

# 2. Add additional browser configurations for our browser-use tool
browser_use_browser_config = {"browser_config": {"headless": False}, "agent_kwargs": {"generate_gif": True}}

# 3. Create the agent, nominating the tool and tool config
web_researcher = WebSurferAgent(
    name="researcher",
    llm_config=llm_config,
    web_tool="browser_use",
    web_tool_kwargs=browser_use_browser_config,
    )

# 4. Run our agent, passing in the tools that our WebSurferAgent has so they can be executed
ag2_news_result = web_researcher.run(
    "Search for the latest news on AG2 AI",
    tools=web_researcher.tools,
)

print(ag2_news_result.summary)

Let’s break it down:

1. Import WebSurferAgent and create an LLM configuration for the browser-use tool to use.

2. We create a configuration dictionary turning off the headless mode (so we can see what’s happening) and saving an animated GIF of the process (shown below).

3. Create the agent, nominating the web tool and passing in the LLM and tool configurations.

4. Run the agent, ensuring we pass the agent’s tools through to the run method so it can add them to the internal executor agent to execute.