Firecrawl

FirecrawlToolkit gives an agent two related tools powered by Firecrawl: web search and single-page scrape. Search can also return each result page's markdown in the same call, so an agent finds and reads sources in one step.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

import os
from ag2 import Agent
from ag2.config import AnthropicConfig
from ag2.extensions.tools.search import FirecrawlToolkit

agent = Agent(
    "researcher",
    config=AnthropicConfig(model="claude-sonnet-5"),
    tools=[FirecrawlToolkit(api_key=os.environ["FIRECRAWL_API_KEY"])],
)

If api_key is omitted, the Firecrawl SDK reads the FIRECRAWL_API_KEY environment variable when a tool runs.

Tools#

Shared defaults#

Constructor defaults are applied to the toolkit's default tools:

1
2
3
4
5
6
7
8

toolkit = FirecrawlToolkit(
    api_key=...,
    limit=5,                  # max results for firecrawl_search
    scrape_results=True,      # include page markdown in firecrawl_search results
    formats=["markdown"],     # output formats for firecrawl_scrape
    only_main_content=True,   # drop navigation, headers, and footers in firecrawl_scrape
    max_characters=20_000,    # cap the page content each tool returns
)

The constructor also accepts api_url (for a self-hosted Firecrawl), http_timeout (seconds, for each HTTP request) and max_retries.

Limiting result size#

A scraped page can be very long, and firecrawl_search with scrape_results=True returns one page per result. Set max_characters to cut each page's markdown (and HTML for firecrawl_scrape) to that many characters. For firecrawl_scrape, links are kept only while their combined length fits. Results that were cut have truncated=True, so the model knows the text is incomplete:

1
2
3
4

toolkit = FirecrawlToolkit(api_key=...)

search_tool = toolkit.search(limit=3, scrape_results=True, max_characters=10_000)
scrape_tool = toolkit.scrape(max_characters=20_000)

max_characters is off by default.

Picking a subset of tools#

Each tool is exposed as a factory method on the toolkit (toolkit.search(), toolkit.scrape()). Call the method to get a ready-to-use tool, then pass only the ones you need to the agent:

1
2
3
4
5
6
7

toolkit = FirecrawlToolkit(api_key=...)

agent = Agent(
    "researcher",
    config=config,
    tools=[toolkit.search(limit=3, scrape_results=True)],
)

Search configuration#

firecrawl_search accepts a required query at execution time. Configure the other options on toolkit.search():

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

toolkit = FirecrawlToolkit(api_key=...)

search_tool = toolkit.search(
    limit=5,                       # results per source; Firecrawl returns 5 when omitted
    sources=["web", "news"],       # "web" | "news"; defaults to web
    categories=["pdf"],            # "github" | "pdf"
    include_domains=["ag2.ai"],    # or exclude_domains=[...]
    tbs="qdr:w",                   # time filter for web results, here the past week
    location="Germany",
    country="DE",
    timeout=30000,                 # Firecrawl API timeout in milliseconds
    scrape_results=True,           # add each result page's markdown
)

agent = Agent("researcher", config=config, tools=[search_tool])

With scrape_results=True, every result also carries the main content of its page as markdown, so the agent does not need a separate scrape call for each result. Each result is then a full page, so keep limit low. scrape_only_main_content and scrape_max_age tune how those pages are read. Leave scrape_results off when titles and descriptions are enough.

Scrape configuration#

firecrawl_scrape accepts a single URL at execution time. Configure the output on toolkit.scrape():

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

toolkit = FirecrawlToolkit(api_key=...)

scrape_tool = toolkit.scrape(
    formats=["markdown", "links"],  # "markdown" | "html" | "links"; defaults to markdown
    only_main_content=True,
    exclude_tags=["nav", "footer"],
    wait_for=1000,                  # milliseconds to wait for the page to render
    max_age=0,                      # 0 always fetches a fresh copy
    timeout=30000,                  # Firecrawl API timeout in milliseconds
)

agent = Agent("researcher", config=config, tools=[scrape_tool])

toolkit.scrape() also accepts include_tags, headers, mobile, proxy, block_ads and store_in_cache.

The search and scrape options (everything except api_key, api_url, http_timeout and max_retries) accept a Variable for deferred context resolution.

Errors#

firecrawl_scrape returns {"error": ...} for a URL that is not http or https, so the model can try again with a valid one. Everything else is raised:

• invalid options, such as setting both include_domains and exclude_domains (raised by toolkit.search() itself, or at call time when the values come from Variables);
• a missing or invalid API key;
• Firecrawl API errors, raised as firecrawl SDK exceptions.

Errors raised while a tool runs reach the model as tool errors.

Result#

firecrawl_search returns a FirecrawlSearchResponse:

markdown is set only when scrape_results=True. In that mode description and snippet come from the page metadata, and news results have no date.

firecrawl_scrape returns a FirecrawlScrapeResult: