Notebooks
Supercharging Web Crawling with Crawl4AI
Use Cases
- Use cases
- Reference Agents
- Notebooks
- All Notebooks
- Solving Multiple Tasks in a Sequence of Chats
- Solving Multiple Tasks in a Sequence of Async Chats
- Using RetrieveChat for Retrieve Augmented Code Generation and Question Answering
- Interactive LLM Agent Dealing with Data Stream
- Agent Chat with Async Human Inputs
- Using RetrieveChat Powered by MongoDB Atlas for Retrieve Augmented Code Generation and Question Answering
- Auto Generated Agent Chat: Task Solving with Langchain Provided Tools as Functions
- Task Solving with Code Generation, Execution and Debugging
- Runtime Logging with AutoGen
- Agent Chat with Multimodal Models: DALLE and GPT-4V
- Using RetrieveChat with Qdrant for Retrieve Augmented Code Generation and Question Answering
- Enhanced Swarm Orchestration with AG2
- Translating Video audio using Whisper and GPT-3.5-turbo
- Agent Tracking with AgentOps
- Group Chat with Coder and Visualization Critic
- ReasoningAgent - Advanced LLM Reasoning with Multiple Search Strategies
- DeepResearchAgent
- Tools with Dependency Injection
- A Uniform interface to call different LLMs
- SQL Agent for Spider text-to-SQL benchmark
- DeepSeek: Adding Browsing Capabilities to AG2
- CaptainAgent
- Adding Browsing Capabilities to AG2
- Auto Generated Agent Chat: Teaching AI New Skills via Natural Language Interaction
- Writing a software application using function calls
- Auto Generated Agent Chat: Function Inception
- Agentic RAG workflow on tabular data from a PDF file
- RealtimeAgent in a Swarm Orchestration
- Using FalkorGraphRagCapability with agents for GraphRAG Question & Answering
- Making OpenAI Assistants Teachable
- Cross-Framework LLM Tool Integration with AG2
- RealtimeAgent with local websocket connection
- Using RetrieveChat Powered by PGVector for Retrieve Augmented Code Generation and Question Answering
- OpenAI Assistants in AutoGen
- Solving Complex Tasks with Nested Chats
- Auto Generated Agent Chat: GPTAssistant with Code Interpreter
- Group Chat with Customized Speaker Selection Method
- Structured output
- Agent Chat with custom model loading
- AutoBuild
- FSM - User can input speaker transition constraints
- Using Neo4j's graph database with AG2 agents for Question & Answering
- Use AutoGen in Databricks with DBRX
- Language Agent Tree Search
- Using a local Telemetry server to monitor a GraphRAG agent
- SocietyOfMindAgent
- Perform Research with Multi-Agent Group Chat
- Group Chat
- StateFlow: Build Workflows through State-Oriented Actions
- Agent Chat with Multimodal Models: LLaVA
- Trip planning with a FalkorDB GraphRAG agent using a Swarm
- Auto Generated Agent Chat: Collaborative Task Solving with Multiple Agents and Human Users
- Agent with memory using Mem0
- Preprocessing Chat History with `TransformMessages`
- AgentOptimizer: An Agentic Way to Train Your LLM Agent
- Using RetrieveChat Powered by Couchbase Capella for Retrieve Augmented Code Generation and Question Answering
- Agent Observability with OpenLIT
- Chatting with a teachable agent
- RealtimeAgent in a Swarm Orchestration
- Conversational Chess using non-OpenAI clients
- Solving Complex Tasks with A Sequence of Nested Chats
- Config loader utility functions
- Currency Calculator: Task Solving with Provided Tools as Functions
- Chat with OpenAI Assistant using function call in AutoGen: OSS Insights for Advanced GitHub Data Analysis
- Web Scraping using Apify Tools
- Agent with memory using Mem0
- Auto Generated Agent Chat: Collaborative Task Solving with Coding and Planning Agent
- Demonstrating the `AgentEval` framework using the task of solving math problems as an example
- RealtimeAgent with WebRTC connection
- Automatically Build Multi-agent System from Agent Library
- Groupchat with Llamaindex agents
- Using Guidance with AutoGen
- RealtimeAgent in a Swarm Orchestration using WebRTC
- Cross-Framework LLM Tool for CaptainAgent
- RealtimeAgent in a Swarm Orchestration
- Websockets: Streaming input and output using websockets
- Auto Generated Agent Chat: Solving Tasks Requiring Web Info
- Auto Generated Agent Chat: Task Solving with Code Generation, Execution, Debugging & Human Feedback
- Assistants with Azure Cognitive Search and Azure Identity
- (Legacy) Implement Swarm-style orchestration with GroupChat
- Auto Generated Agent Chat: Task Solving with Provided Tools as Functions
- Agentchat MathChat
- Small, Local Model (IBM Granite) Multi-Agent RAG
- Group Chat with Retrieval Augmented Generation
- Run a standalone AssistantAgent
- Nested Chats for Tool Use in Conversational Chess
- Swarm Orchestration with AG2
- OptiGuide with Nested Chats in AutoGen
- Usage tracking with AutoGen
- From Dad Jokes To Sad Jokes: Function Calling with GPTAssistantAgent
- Discord, Slack, and Telegram messaging tools
- RAG OpenAI Assistants in AutoGen
- Auto Generated Agent Chat: Group Chat with GPTAssistantAgent
- Task Solving with Provided Tools as Functions (Asynchronous Function Calls)
- Generate Dalle Images With Conversable Agents
- Chat Context Dependency Injection
- Solving Multiple Tasks in a Sequence of Chats with Different Conversable Agent Pairs
- Engaging with Multimodal Models: GPT-4V in AutoGen
- Mitigating Prompt hacking with JSON Mode in Autogen
- Using Neo4j's native GraphRAG SDK with AG2 agents for Question & Answering
- Supercharging Web Crawling with Crawl4AI
- WebSurferAgent
- RealtimeAgent with gemini client
- Community Gallery
Notebooks
Supercharging Web Crawling with Crawl4AI
Supercharging Web Crawling with Crawl4AI
Installation
To get started with the crawl4ai
integration in AG2, follow these
steps:
-
Install AG2 with the
crawl4ai
extra:pip install ag2[crawl4ai]
Note: If you have been using
autogen
orpyautogen
, all you need to do is upgrade it using:pip install -U autogen[crawl4ai]
or
pip install -U pyautogen[crawl4ai]
as
pyautogen
,autogen
, andag2
are aliases for the same PyPI package. -
Set up Playwright:
# Installs Playwright and browsers for all OS playwright install # Additional command, mandatory for Linux only playwright install-deps
-
For running the code in Jupyter, use
nest_asyncio
to allow nested event loops.bash pip install nest_asyncio
You’re all set! Now you can start using browsing features in AG2.
Imports
import os
import nest_asyncio
from pydantic import BaseModel
from autogen import AssistantAgent, UserProxyAgent
from autogen.tools.experimental import Crawl4AITool
nest_asyncio.apply()
LLM-Free Crawl4AI
config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]
llm_config = {
"config_list": config_list,
}
user_proxy = UserProxyAgent(name="user_proxy", human_input_mode="NEVER")
assistant = AssistantAgent(name="assistant", llm_config=llm_config)
crawlai_tool = Crawl4AITool()
crawlai_tool.register_for_execution(user_proxy)
crawlai_tool.register_for_llm(assistant)
result = user_proxy.initiate_chat(
recipient=assistant,
message="Get info from https://docs.ag2.ai/docs/Home",
max_turns=2,
)
Crawl4AI with LLM
Note:
Crawl4AI
is built on top of LiteLLM and supports the same models as LiteLLM.We had great experience with
OpenAI
,Anthropic
,Gemini
andOllama
. However, as of this writing,DeepSeek
is encountering some issues.
config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]
llm_config = {
"config_list": config_list,
}
user_proxy = UserProxyAgent(name="user_proxy", human_input_mode="NEVER")
assistant = AssistantAgent(name="assistant", llm_config=llm_config)
# Set llm_config to Crawl4AITool
crawlai_tool = Crawl4AITool(llm_config=llm_config)
crawlai_tool.register_for_execution(user_proxy)
crawlai_tool.register_for_llm(assistant)
result = user_proxy.initiate_chat(
recipient=assistant,
message="Get info from https://docs.ag2.ai/docs/Home",
max_turns=2,
)
Crawl4AI with LLM & Schema for Structured Data
config_list = [{"model": "gpt-4o-mini", "api_key": os.environ["OPENAI_API_KEY"]}]
llm_config = {
"config_list": config_list,
}
user_proxy = UserProxyAgent(name="user_proxy", human_input_mode="NEVER")
assistant = AssistantAgent(name="assistant", llm_config=llm_config)
class Blog(BaseModel):
title: str
url: str
# Set llm_config and extraction_model to Crawl4AITool
crawlai_tool = Crawl4AITool(llm_config=llm_config, extraction_model=Blog)
crawlai_tool.register_for_execution(user_proxy)
crawlai_tool.register_for_llm(assistant)
message = "Extract all blog posts from https://docs.ag2.ai/blog"
result = user_proxy.initiate_chat(
recipient=assistant,
message=message,
max_turns=2,
)