Structured output#

LLM providers offer functionality for defining a structure of the messages generated by LLMs, AG2 enables this functionality by propagating a response_format, in the LLM configuration for your agents, to the underlying client.

Structured outputs are available for a number of Model Providers, see the Supported model providers section below. In this example we will use OpenAI as the model provider.

pip install -U ag2[openai]

Supported model providers#

AG2 has structured output support for following model providers:

• OpenAI (openai)

• Anthropic (anthropic)

• Google Gemini (google)

• Ollama (ollama)

Set your API Endpoint#

The LLMConfig.from_json function loads a list of configurations from an environment variable or a json file. Structured Output is supported by OpenAI’s models from gpt-4-0613 and gpt-3.5-turbo-0613.

import autogen

Example: math reasoning#

Using structured output, we can enforce chain-of-thought reasoning in the model to output an answer in a structured, step-by-step way.

Define the reasoning model#

First we will define the math reasoning model. This model will indirectly force the LLM to solve the posed math problems iteratively through math reasoning steps.

from pydantic import BaseModel

class Step(BaseModel):
    explanation: str
    output: str

class MathReasoning(BaseModel):
    steps: list[Step]
    final_answer: str

Define chat actors#

Now we can define the agents that will solve the posed math problem. We will keep this example simple; we will use a UserProxyAgent to input the math problem and an AssistantAgent to solve it.

The AssistantAgent will be constrained to solving the math problem step-by-step by using the MathReasoning response format we defined above.

The response_format is added to the LLM configuration and then this configuration is applied to the agent.

llm_config = autogen.LLMConfig.from_json(
    path="OAI_CONFIG_LIST",
    cache_seed=42,
    response_format=MathReasoning,
).where(tags=["gpt-4o", "gpt-4o-mini"])

user_proxy = autogen.UserProxyAgent(
    name="User_proxy",
    system_message="A human admin.",
    human_input_mode="NEVER",
)

assistant = autogen.AssistantAgent(
    name="Math_solver",
    llm_config=llm_config,  # Response Format is in the configuration
)

Start the chat#

Let’s now start the chat and prompt the assistant to solve a simple equation. The assistant agent should return a response solving the equation using a step-by-step MathReasoning model.

user_proxy.initiate_chat(assistant, message="how can I solve 8x + 7 = -23", max_turns=1, summary_method="last_msg")

Formatting a response#

When defining a response_format, you have the flexibility to customize how the output is parsed and presented, making it more user-friendly. To demonstrate this, we’ll add a format method to our MathReasoning model. This method will define the logic for transforming the raw JSON response into a more human-readable and accessible format.

Define the reasoning model#

Let’s redefine the MathReasoning model to include a format method. This method will allow the underlying client to parse the return value from the LLM into a more human-readable format. If the format method is not defined, the client will default to returning the model’s JSON representation, as demonstrated in the previous example.

from pydantic import BaseModel

class Step(BaseModel):
    explanation: str
    output: str

class MathReasoning(BaseModel):
    steps: list[Step]
    final_answer: str

    def format(self) -> str:
        steps_output = "\n".join(
            f"Step {i + 1}: {step.explanation}\n  Output: {step.output}" for i, step in enumerate(self.steps)
        )
        return f"{steps_output}\n\nFinal Answer: {self.final_answer}"

Define chat actors and start the chat#

The rest of the process is the same as in the previous example: define the actors and start the chat.

Observe how the Math_solver agent now communicates using the format we have defined in our MathReasoning.format method.

for config in llm_config.config_list:
    config.response_format = MathReasoning

user_proxy = autogen.UserProxyAgent(
    name="User_proxy",
    system_message="A human admin.",
    human_input_mode="NEVER",
)

assistant = autogen.AssistantAgent(
    name="Math_solver",
    llm_config=llm_config,
)

print(
    user_proxy.initiate_chat(
        assistant, message="how can I solve 8x + 7 = -23", max_turns=1, summary_method="last_msg"
    ).summary
)

Normal function calling still works alongside structured output, so your agent can have a response format while still calling tools.

@user_proxy.register_for_execution()
@assistant.register_for_llm(description="You can use this function call to solve addition")
def add(x: int, y: int) -> int:
    return x + y

user_proxy.initiate_chat(
    assistant, message="solve 3 + 4 by calling appropriate function", max_turns=2, summary_method="last_msg"
)