Use Cases
- Use cases
- Reference Agents
- Notebooks
- All Notebooks
- Group Chat with Customized Speaker Selection Method
- RAG OpenAI Assistants in AutoGen
- Using RetrieveChat with Qdrant for Retrieve Augmented Code Generation and Question Answering
- Auto Generated Agent Chat: Function Inception
- Task Solving with Provided Tools as Functions (Asynchronous Function Calls)
- Using Guidance with AutoGen
- Solving Complex Tasks with A Sequence of Nested Chats
- Group Chat
- Solving Multiple Tasks in a Sequence of Async Chats
- Auto Generated Agent Chat: Task Solving with Provided Tools as Functions
- Conversational Chess using non-OpenAI clients
- RealtimeAgent with local websocket connection
- Web Scraping using Apify Tools
- DeepSeek: Adding Browsing Capabilities to AG2
- Interactive LLM Agent Dealing with Data Stream
- Generate Dalle Images With Conversable Agents
- Supercharging Web Crawling with Crawl4AI
- RealtimeAgent in a Swarm Orchestration
- Perform Research with Multi-Agent Group Chat
- Agent Tracking with AgentOps
- Translating Video audio using Whisper and GPT-3.5-turbo
- Automatically Build Multi-agent System from Agent Library
- Auto Generated Agent Chat: Collaborative Task Solving with Multiple Agents and Human Users
- Structured output
- CaptainAgent
- Group Chat with Coder and Visualization Critic
- Cross-Framework LLM Tool Integration with AG2
- Using FalkorGraphRagCapability with agents for GraphRAG Question & Answering
- Demonstrating the `AgentEval` framework using the task of solving math problems as an example
- RealtimeAgent in a Swarm Orchestration using WebRTC
- A Uniform interface to call different LLMs
- From Dad Jokes To Sad Jokes: Function Calling with GPTAssistantAgent
- Solving Complex Tasks with Nested Chats
- Usage tracking with AutoGen
- Agent with memory using Mem0
- Using RetrieveChat Powered by PGVector for Retrieve Augmented Code Generation and Question Answering
- Tools with Dependency Injection
- Solving Multiple Tasks in a Sequence of Chats with Different Conversable Agent Pairs
- WebSurferAgent
- Using RetrieveChat Powered by MongoDB Atlas for Retrieve Augmented Code Generation and Question Answering
- Assistants with Azure Cognitive Search and Azure Identity
- ReasoningAgent - Advanced LLM Reasoning with Multiple Search Strategies
- Agentic RAG workflow on tabular data from a PDF file
- Making OpenAI Assistants Teachable
- Run a standalone AssistantAgent
- AutoBuild
- Solving Multiple Tasks in a Sequence of Chats
- Currency Calculator: Task Solving with Provided Tools as Functions
- Swarm Orchestration with AG2
- Use AutoGen in Databricks with DBRX
- Using a local Telemetry server to monitor a GraphRAG agent
- Auto Generated Agent Chat: Solving Tasks Requiring Web Info
- StateFlow: Build Workflows through State-Oriented Actions
- Groupchat with Llamaindex agents
- Using Neo4j's native GraphRAG SDK with AG2 agents for Question & Answering
- Agent Chat with Multimodal Models: LLaVA
- Group Chat with Retrieval Augmented Generation
- Runtime Logging with AutoGen
- SocietyOfMindAgent
- Agent Chat with Multimodal Models: DALLE and GPT-4V
- Agent Observability with OpenLIT
- Mitigating Prompt hacking with JSON Mode in Autogen
- Trip planning with a FalkorDB GraphRAG agent using a Swarm
- Language Agent Tree Search
- Auto Generated Agent Chat: Collaborative Task Solving with Coding and Planning Agent
- OptiGuide with Nested Chats in AutoGen
- Auto Generated Agent Chat: Task Solving with Langchain Provided Tools as Functions
- Writing a software application using function calls
- Auto Generated Agent Chat: GPTAssistant with Code Interpreter
- Adding Browsing Capabilities to AG2
- Agentchat MathChat
- Chatting with a teachable agent
- RealtimeAgent with gemini client
- Preprocessing Chat History with `TransformMessages`
- Chat with OpenAI Assistant using function call in AutoGen: OSS Insights for Advanced GitHub Data Analysis
- Websockets: Streaming input and output using websockets
- Task Solving with Code Generation, Execution and Debugging
- Agent Chat with Async Human Inputs
- Agent Chat with custom model loading
- Chat Context Dependency Injection
- Nested Chats for Tool Use in Conversational Chess
- Auto Generated Agent Chat: Group Chat with GPTAssistantAgent
- Cross-Framework LLM Tool for CaptainAgent
- Auto Generated Agent Chat: Teaching AI New Skills via Natural Language Interaction
- SQL Agent for Spider text-to-SQL benchmark
- Auto Generated Agent Chat: Task Solving with Code Generation, Execution, Debugging & Human Feedback
- OpenAI Assistants in AutoGen
- (Legacy) Implement Swarm-style orchestration with GroupChat
- Enhanced Swarm Orchestration with AG2
- Using RetrieveChat for Retrieve Augmented Code Generation and Question Answering
- Using Neo4j's graph database with AG2 agents for Question & Answering
- AgentOptimizer: An Agentic Way to Train Your LLM Agent
- Engaging with Multimodal Models: GPT-4V in AutoGen
- RealtimeAgent with WebRTC connection
- FSM - User can input speaker transition constraints
- Config loader utility functions
- Community Gallery
FSM - User can input speaker transition constraints
Explore the demonstration of the Finite State Machine implementation, which allows the user to input speaker transition constraints.
AutoGen offers conversable agents powered by LLM, tool, or human, which can be used to perform tasks collectively via automated chat. This framework allows tool use and human participation through multi-agent conversation. Please find documentation about this feature here.
This notebook is about using graphs to define the transition paths amongst speakers.
Benefits - This contribution fills the gap between the current modes of GroupChat Class (auto, manual, round_robin) and an expressive directed graph. See Motivation for more detailed discussion.
Install pyautogen
:
pip install pyautogen
For more information, please refer to the installation guide.
%%capture --no-stderr
%pip install pyautogen[graph]>=0.2.11
import random # noqa E402
import matplotlib.pyplot as plt
import networkx as nx
import autogen
from autogen.agentchat.conversable_agent import ConversableAgent
from autogen.agentchat.assistant_agent import AssistantAgent
from autogen.agentchat.groupchat import GroupChat
from autogen.graph_utils import visualize_speaker_transitions_dict
print(autogen.__version__)
0.2.25
Motivation
The current GroupChat class allows transitioning to any agent (with or without the decision of the LLM), some use cases might demand for more control over transition. A graph is a possible way to control the transition paths, where each node represents an agent and each directed edge represents possible transition paths. Let’s illustrate the current transition paths for a GroupChat with five agents.
config_list_gpt4 = {
"timeout": 600,
"cache_seed": 44, # change the seed for different trials
"config_list": autogen.config_list_from_json(
"OAI_CONFIG_LIST",
filter_dict={"tags": ["gpt-4", "gpt-4-32k"]}, # comment out to get all
),
"temperature": 0,
}
agents = [ConversableAgent(name=f"Agent{i}", llm_config=False) for i in range(5)]
allowed_speaker_transitions_dict = {agent: [other_agent for other_agent in agents] for agent in agents}
visualize_speaker_transitions_dict(allowed_speaker_transitions_dict, agents)
Possibly interesting transition paths
- Hub and Spoke
- Sequential Team Operations
- Think aloud and debate
agents = [ConversableAgent(name=f"Agent{i}", llm_config=False) for i in range(5)]
allowed_speaker_transitions_dict = {
agents[0]: [agents[1], agents[2], agents[3], agents[4]],
agents[1]: [agents[0]],
agents[2]: [agents[0]],
agents[3]: [agents[0]],
agents[4]: [agents[0]],
}
visualize_speaker_transitions_dict(allowed_speaker_transitions_dict, agents)
# Sequential Team Operations
# Create an empty directed graph
speaker_transitions_dict = {}
teams = ["A", "B", "C"]
team_size = 5
def get_agent_of_name(agents, name) -> ConversableAgent:
for agent in agents:
if agent.name == name:
return agent
# Create a list of 15 agents 3 teams x 5 agents
agents = [ConversableAgent(name=f"{team}{i}", llm_config=False) for team in teams for i in range(team_size)]
# Loop through each team and add members and their connections
for team in teams:
for i in range(team_size):
member = f"{team}{i}"
# Connect each member to other members of the same team
speaker_transitions_dict[get_agent_of_name(agents, member)] = [
get_agent_of_name(agents, name=f"{team}{j}") for j in range(team_size) if j != i
]
# Team leaders connection
print(get_agent_of_name(agents, name="B0"))
speaker_transitions_dict[get_agent_of_name(agents, "A0")].append(get_agent_of_name(agents, name="B0"))
speaker_transitions_dict[get_agent_of_name(agents, "B0")].append(get_agent_of_name(agents, name="C0"))
visualize_speaker_transitions_dict(speaker_transitions_dict, agents)
<autogen.agentchat.conversable_agent.ConversableAgent object at 0x7f605e116260>
agents = [ConversableAgent(name=f"Agent{i}", llm_config=False) for i in range(2)]
allowed_speaker_transitions_dict = {
agents[0]: [agents[0], agents[1]],
agents[1]: [agents[0], agents[1]],
}
visualize_speaker_transitions_dict(allowed_speaker_transitions_dict, agents)
Demonstration
GroupChat
now takes in two optional arguments. -
allowed_or_disallowed_speaker_transitions: The keys are source agents,
and the values are agents that the key agent can/can’t transit to,
depending on speaker_transitions_type. Default is None, which means all
agents can transit to all other agents. - speaker_transitions_type:
whether the speaker_transitions_type is a dictionary containing lists of
allowed agents or disallowed agents. “allowed” means the
allowed_or_disallowed_speaker_transitions
is a dictionary containing
lists of allowed agents.
Team Operations
# Create an empty directed graph
agents = []
speaker_transitions_dict = {}
secret_values = {}
# Outer loop for prefixes 'A', 'B', 'C'
for prefix in ["A", "B", "C"]:
# Add 3 nodes with each prefix to the graph using a for loop
for i in range(3):
node_id = f"{prefix}{i}"
secret_value = random.randint(1, 5) # Generate a random secret value
secret_values[node_id] = secret_value
# Create an AssistantAgent for each node (assuming AssistantAgent is a defined class)
agents.append(
AssistantAgent(
name=node_id,
system_message=f"""Your name is {node_id}.
Do not respond as the speaker named in the NEXT tag if your name is not in the NEXT tag. Instead, suggest a relevant team leader to handle the mis-tag, with the NEXT: tag.
You have {secret_value} chocolates.
The list of players are [A0, A1, A2, B0, B1, B2, C0, C1, C2].
Your first character of your name is your team, and your second character denotes that you are a team leader if it is 0.
CONSTRAINTS: Team members can only talk within the team, whilst team leader can talk to team leaders of other teams but not team members of other teams.
You can use NEXT: to suggest the next speaker. You have to respect the CONSTRAINTS, and can only suggest one player from the list of players, i.e., do not suggest A3 because A3 is not from the list of players.
Team leaders must make sure that they know the sum of the individual chocolate count of all three players in their own team, i.e., A0 is responsible for team A only.
Keep track of the player's tally using a JSON format so that others can check the total tally. Use
A0:?, A1:?, A2:?,
B0:?, B1:?, B2:?,
C0:?, C1:?, C2:?
If you are the team leader, you should aggregate your team's total chocolate count to cooperate.
Once the team leader know their team's tally, they can suggest another team leader for them to find their team tally, because we need all three team tallys to succeed.
Use NEXT: to suggest the next speaker, e.g., NEXT: A0.
Once we have the total tally from all nine players, sum up all three teams' tally, then terminate the discussion using TERMINATE.
""",
llm_config=config_list_gpt4,
)
)
speaker_transitions_dict[agents[-1]] = []
# Add edges between nodes with the same prefix using a nested for loop
for source_node in range(3):
source_id = f"{prefix}{source_node}"
for target_node in range(3):
target_id = f"{prefix}{target_node}"
if source_node != target_node: # To avoid self-loops
speaker_transitions_dict[get_agent_of_name(agents, source_id)].append(
get_agent_of_name(agents, name=target_id)
)
# Adding edges between teams
speaker_transitions_dict[get_agent_of_name(agents, "A0")].append(get_agent_of_name(agents, name="B0"))
speaker_transitions_dict[get_agent_of_name(agents, "A0")].append(get_agent_of_name(agents, name="C0"))
speaker_transitions_dict[get_agent_of_name(agents, "B0")].append(get_agent_of_name(agents, name="A0"))
speaker_transitions_dict[get_agent_of_name(agents, "B0")].append(get_agent_of_name(agents, name="C0"))
speaker_transitions_dict[get_agent_of_name(agents, "C0")].append(get_agent_of_name(agents, name="A0"))
speaker_transitions_dict[get_agent_of_name(agents, "C0")].append(get_agent_of_name(agents, name="B0"))
# Visualization only
graph = nx.DiGraph()
# Add nodes
graph.add_nodes_from([agent.name for agent in agents])
# Add edges
for key, value in speaker_transitions_dict.items():
for agent in value:
graph.add_edge(key.name, agent.name)
# Visualize
# Draw the graph with secret values annotated
plt.figure(figsize=(12, 10))
pos = nx.spring_layout(graph) # positions for all nodes
# Draw nodes with their colors
nx.draw(graph, pos, with_labels=True, font_weight="bold")
# Annotate secret values
for node, (x, y) in pos.items():
secret_value = secret_values[node]
plt.text(x, y + 0.1, s=f"Secret: {secret_value}", horizontalalignment="center")
plt.show()
# Termination message detection
def is_termination_msg(content) -> bool:
have_content = content.get("content", None) is not None
return have_content and "TERMINATE" in content["content"]
# Terminates the conversation when TERMINATE is detected.
user_proxy = autogen.UserProxyAgent(
name="User_proxy",
system_message="Terminator admin.",
code_execution_config=False,
is_termination_msg=is_termination_msg,
human_input_mode="NEVER",
)
agents.append(user_proxy)
group_chat = GroupChat(
agents=agents,
messages=[],
max_round=20,
allowed_or_disallowed_speaker_transitions=speaker_transitions_dict,
speaker_transitions_type="allowed",
)
# Create the manager
manager = autogen.GroupChatManager(
groupchat=group_chat,
llm_config=config_list_gpt4,
code_execution_config=False,
is_termination_msg=is_termination_msg,
)
# Initiates the chat with Alice
agents[0].initiate_chat(
manager,
message="""
There are 9 players in this game, split equally into Teams A, B, C. Therefore each team has 3 players, including the team leader.
The task is to find out the sum of chocolate count from all nine players. I will now start with my team.
NEXT: A1""",
)
WARNING:root:Warning: There are isolated agent nodes, there are not incoming nor outgoing edges. Isolated agents: ['User_proxy']
WARNING:root:Warning: The set of agents in allowed_speaker_transitions do not match agents. Offending agents: ['User_proxy']
A0 (to chat_manager):
There are 9 players in this game, split equally into Teams A, B, C. Therefore each team has 3 players, including the team leader.
The task is to find out the sum of chocolate count from all nine players. I will now start with my team.
NEXT: A1
--------------------------------------------------------------------------------
A1 (to chat_manager):
As the team leader of Team A, it's my responsibility to gather the chocolate count from my team members. I have 1 chocolate.
A0:?, A1:1, A2:?,
B0:?, B1:?, B2:?,
C0:?, C1:?, C2:?
NEXT: A2
--------------------------------------------------------------------------------
A2 (to chat_manager):
I have 5 chocolates.
A0:?, A1:1, A2:5,
B0:?, B1:?, B2:?,
C0:?, C1:?, C2:?
NEXT: A0
--------------------------------------------------------------------------------
A0 (to chat_manager):
I have 1 chocolate. Now that I have the counts from A1 and A2, I can calculate our team's total.
A0:1, A1:1, A2:5,
B0:?, B1:?, B2:?,
C0:?, C1:?, C2:?
Team A's total tally is 7 chocolates. It's time to find out the tally from another team. I'll pass this to the next team leader.
NEXT: B0
--------------------------------------------------------------------------------
B0 (to chat_manager):
As the team leader of Team B, it's my responsibility to gather the chocolate count from my team members. I have 1 chocolate.
A0:1, A1:1, A2:5,
B0:1, B1:?, B2:?,
C0:?, C1:?, C2:?
Now, I need to find out the chocolate counts from B1 and B2.
NEXT: B1
--------------------------------------------------------------------------------
B1 (to chat_manager):
I have 3 chocolates.
A0:1, A1:1, A2:5,
B0:1, B1:3, B2:?,
C0:?, C1:?, C2:?
NEXT: B2
--------------------------------------------------------------------------------
B2 (to chat_manager):
I have 4 chocolates. Now that I have the counts from B0 and B1, I can calculate our team's total.
A0:1, A1:1, A2:5,
B0:1, B1:3, B2:4,
C0:?, C1:?, C2:?
Team B's total tally is 8 chocolates. It's time to find out the tally from the last team. I'll pass this to the next team leader.
NEXT: C0
--------------------------------------------------------------------------------
B0 (to chat_manager):
I'm sorry for any confusion, but it seems there's been a misunderstanding in the flow of communication. As B0, I should be the one to summarize the tally for Team B and pass the conversation to the next team leader. Let me correct that.
A0:1, A1:1, A2:5,
B0:1, B1:3, B2:4,
C0:?, C1:?, C2:?
With the counts from B1 and B2, Team B's total tally is indeed 8 chocolates. Now, it's time to find out the tally from Team C. I'll pass this to the next team leader.
NEXT: C0
--------------------------------------------------------------------------------
C0 (to chat_manager):
As the team leader of Team C, it's my responsibility to gather the chocolate count from my team members. I have 3 chocolates.
A0:1, A1:1, A2:5,
B0:1, B1:3, B2:4,
C0:3, C1:?, C2:?
Now, I need to find out the chocolate counts from C1 and C2.
NEXT: C1
--------------------------------------------------------------------------------
C1 (to chat_manager):
I have 3 chocolates.
A0:1, A1:1, A2:5,
B0:1, B1:3, B2:4,
C0:3, C1:3, C2:?
NEXT: C2
--------------------------------------------------------------------------------
C2 (to chat_manager):
I have 2 chocolates. Now that I have the counts from C0 and C1, I can calculate our team's total.
A0:1, A1:1, A2:5,
B0:1, B1:3, B2:4,
C0:3, C1:3, C2:2
Team C's total tally is 8 chocolates. Now that we have the tallies from all teams, we can sum them up.
Team A's total is 7 chocolates, Team B's total is 8 chocolates, and Team C's total is 8 chocolates. The grand total is 23 chocolates.
TERMINATE
--------------------------------------------------------------------------------
ChatResult(chat_id=None, chat_history=[{'content': '\n There are 9 players in this game, split equally into Teams A, B, C. Therefore each team has 3 players, including the team leader.\n The task is to find out the sum of chocolate count from all nine players. I will now start with my team.\n NEXT: A1', 'role': 'assistant'}, {'content': "As the team leader of Team A, it's my responsibility to gather the chocolate count from my team members. I have 1 chocolate. \n\nA0:?, A1:1, A2:?,\n\nB0:?, B1:?, B2:?,\n\nC0:?, C1:?, C2:?\n\nNEXT: A2", 'name': 'A1', 'role': 'user'}, {'content': 'I have 5 chocolates.\n\nA0:?, A1:1, A2:5,\n\nB0:?, B1:?, B2:?,\n\nC0:?, C1:?, C2:?\n\nNEXT: A0', 'name': 'A2', 'role': 'user'}, {'content': "I have 1 chocolate. Now that I have the counts from A1 and A2, I can calculate our team's total.\n\nA0:1, A1:1, A2:5,\n\nB0:?, B1:?, B2:?,\n\nC0:?, C1:?, C2:?\n\nTeam A's total tally is 7 chocolates. It's time to find out the tally from another team. I'll pass this to the next team leader.\n\nNEXT: B0", 'role': 'assistant'}, {'content': "As the team leader of Team B, it's my responsibility to gather the chocolate count from my team members. I have 1 chocolate.\n\nA0:1, A1:1, A2:5,\n\nB0:1, B1:?, B2:?,\n\nC0:?, C1:?, C2:?\n\nNow, I need to find out the chocolate counts from B1 and B2.\n\nNEXT: B1", 'name': 'B0', 'role': 'user'}, {'content': 'I have 3 chocolates.\n\nA0:1, A1:1, A2:5,\n\nB0:1, B1:3, B2:?,\n\nC0:?, C1:?, C2:?\n\nNEXT: B2', 'name': 'B1', 'role': 'user'}, {'content': "I have 4 chocolates. Now that I have the counts from B0 and B1, I can calculate our team's total.\n\nA0:1, A1:1, A2:5,\n\nB0:1, B1:3, B2:4,\n\nC0:?, C1:?, C2:?\n\nTeam B's total tally is 8 chocolates. It's time to find out the tally from the last team. I'll pass this to the next team leader.\n\nNEXT: C0", 'name': 'B2', 'role': 'user'}, {'content': "I'm sorry for any confusion, but it seems there's been a misunderstanding in the flow of communication. As B0, I should be the one to summarize the tally for Team B and pass the conversation to the next team leader. Let me correct that.\n\nA0:1, A1:1, A2:5,\n\nB0:1, B1:3, B2:4,\n\nC0:?, C1:?, C2:?\n\nWith the counts from B1 and B2, Team B's total tally is indeed 8 chocolates. Now, it's time to find out the tally from Team C. I'll pass this to the next team leader.\n\nNEXT: C0", 'name': 'B0', 'role': 'user'}, {'content': "As the team leader of Team C, it's my responsibility to gather the chocolate count from my team members. I have 3 chocolates.\n\nA0:1, A1:1, A2:5,\n\nB0:1, B1:3, B2:4,\n\nC0:3, C1:?, C2:?\n\nNow, I need to find out the chocolate counts from C1 and C2.\n\nNEXT: C1", 'name': 'C0', 'role': 'user'}, {'content': 'I have 3 chocolates.\n\nA0:1, A1:1, A2:5,\n\nB0:1, B1:3, B2:4,\n\nC0:3, C1:3, C2:?\n\nNEXT: C2', 'name': 'C1', 'role': 'user'}, {'content': "I have 2 chocolates. Now that I have the counts from C0 and C1, I can calculate our team's total.\n\nA0:1, A1:1, A2:5,\n\nB0:1, B1:3, B2:4,\n\nC0:3, C1:3, C2:2\n\nTeam C's total tally is 8 chocolates. Now that we have the tallies from all teams, we can sum them up.\n\nTeam A's total is 7 chocolates, Team B's total is 8 chocolates, and Team C's total is 8 chocolates. The grand total is 23 chocolates.\n\nTERMINATE", 'name': 'C2', 'role': 'user'}], summary="I have 2 chocolates. Now that I have the counts from C0 and C1, I can calculate our team's total.\n\nA0:1, A1:1, A2:5,\n\nB0:1, B1:3, B2:4,\n\nC0:3, C1:3, C2:2\n\nTeam C's total tally is 8 chocolates. Now that we have the tallies from all teams, we can sum them up.\n\nTeam A's total is 7 chocolates, Team B's total is 8 chocolates, and Team C's total is 8 chocolates. The grand total is 23 chocolates.\n\n", cost={'usage_including_cached_inference': {'total_cost': 0.5525399999999999, 'gpt-4': {'cost': 0.5525399999999999, 'prompt_tokens': 18174, 'completion_tokens': 122, 'total_tokens': 18296}}, 'usage_excluding_cached_inference': {'total_cost': 0.5525399999999999, 'gpt-4': {'cost': 0.5525399999999999, 'prompt_tokens': 18174, 'completion_tokens': 122, 'total_tokens': 18296}}}, human_input=[])