Model Configuration#
The AG2 framework provides an explicit, predictable, and type-safe way to configure Large Language Models (LLMs) for your agents. The configuration API is designed to provide a consistent developer experience across different model providers while maintaining strong typing support.
Supported Providers#
AG2 supports multiple LLM providers through dedicated configuration classes. Each provider requires its respective optional dependencies to be installed.
| Provider | Configuration Class | Installation Command |
|---|---|---|
| OpenAI Responses | OpenAIResponsesConfig | pip install "ag2[openai]" |
| OpenAI Chat Completions | OpenAICompletionsConfig | pip install "ag2[openai]" |
| OpenAI Decisions | OpenAIDecisionsConfig | pip install "ag2[openai]" |
| Anthropic | AnthropicConfig | pip install "ag2[anthropic]" |
| Gemini | GeminiConfig | pip install "ag2[gemini]" |
| Gemini on Vertex AI | VertexAIConfig | pip install "ag2[gemini]" |
| Amazon Bedrock | BedrockConfig | pip install "ag2[bedrock]" |
| Ollama | OllamaConfig | pip install "ag2[ollama]" |
| DashScope | DashScopeConfig | pip install "ag2[dashscope]" |
| xAI | XAIConfig | pip install "ag2[xai]" |
| Z.AI | ZAIConfig | pip install "ag2[zai]" |
| Mistral | MistralConfig | pip install "ag2[mistral]" |
| TypeSafe (Jev) | TypeSafeConfig | pip install "ag2[typesafe]" |
Use OpenAIResponsesConfig for new OpenAI integrations. Use OpenAICompletionsConfig for the Chat Completions API and compatible endpoints.
OpenAIConfig changes in 1.4.0
OpenAIConfig still uses Chat Completions, but this compatibility name is deprecated. Its create() method emits a DeprecationWarning; constructing or copying the config does not. Use OpenAICompletionsConfig explicitly to preserve Chat Completions behavior, or migrate to OpenAIResponsesConfig for the Responses API. In 1.4.0, the deprecated Chat Completions binding will be removed and OpenAIConfig will become an alias for OpenAIResponsesConfig. Chat Completions parameters such as max_completion_tokens and reasoning_effort must be translated to Responses parameters such as max_output_tokens and reasoning when migrating.
How to Configure a Model#
Basic Configuration#
To configure a model, import the specific provider's configuration class and initialize it with your desired parameters. The most common parameters are model, api_key, and base_url.
ag2[anthropic] requires anthropic>=1.15.0,<3. See Anthropic Configuration for the httpx2 HTTP client and why temperature, top_p and top_k are deprecated.
Credentials follow the standard AWS resolution chain: explicit aws_access_key_id / aws_secret_access_key, a named profile_name, environment variables, shared config files, or instance roles. model accepts a Bedrock model id or an inference-profile ARN. See Amazon Bedrock authentication for the API-key (bearer token) alternative.
The Decisions API answers a typed question instead of generating text, so the agent must set a decision response_schema. See OpenAI Decisions Configuration.
Jev answers a typed question instead of generating text, so the agent must set a decision response_schema. See TypeSafe (Jev) Configuration.
Tip
AG2 is designed to be async and streaming-first, so for the best user experience it is recommended to enable streaming on the model provider configurations for models that support it. As shown above, Streaming has been set to True in each config that supports it; TypeSafe's Jev and OpenAI's Decisions API answer in a single response and have no streaming option.
Using Environment Variables#
For security and convenience, you don't need to hardcode your API keys. If api_key is not explicitly provided, the configuration will automatically attempt to load it from your environment variables.
The system looks for provider-specific keys (e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY, XAI_API_KEY, ZAI_API_KEY, TYPESAFE_API_KEY).
Controlling OpenAI Reasoning#
OpenAI reasoning models spend extra tokens thinking before they answer. Both OpenAI configs let you bound that effort, and the Responses config can also request a summary of the thinking. See the OpenAI reasoning guide for which values each model accepts.
OpenAICompletionsConfig (Chat Completions) takes reasoning_effort. OpenAIResponsesConfig takes the Responses API reasoning object:
effort is one of none, minimal, low, medium, high or xhigh; the models that accept each value differ. summary is one of auto, concise or detailed.
The model only returns its thinking when you set summary; without it the model still reasons, but no thinking text reaches the stream. The summary is emitted as ModelReasoning events (OpenAIReasoningEvent, a subclass), and reasoning tokens are reported on ModelResponse.usage.thinking_tokens.
OpenAI Prompt Cache Retention#
Set ttl in OpenAIResponsesConfig's prompt_cache_options to "12h" to keep cache entries for at least 12 hours instead of the default 30 minutes (gpt-5.6 and later). OpenAICompletionsConfig takes the same option, but Chat Completions accepts only "30m" and rejects "12h". See the OpenAI prompt caching guide for which models support each lifetime:
Cache writes are reported on ModelResponse.usage.cache_creation_input_tokens. The Responses API also reports how many of those tokens were written with 12-hour retention, which ag2 exposes as usage.cache_creation_extended_input_tokens; 12-hour writes are priced differently, so cost accounting can tell them apart. Anthropic reports its 1-hour cache writes in the same field (see Anthropic Prompt Cache Retention).
Anthropic Configuration#
AnthropicConfig connects to the Claude API through the official anthropic SDK. Install the optional dependency with pip install "ag2[anthropic]", which requires anthropic>=1.15.0,<3.
If api_key is not passed explicitly, it is resolved from the ANTHROPIC_API_KEY environment variable.
Anthropic Prompt Cache Retention#
Anthropic cache entries live for 5 minutes by default. To keep a large document cached for an hour, set ttl to "1h" in its cache_control vendor metadata:
1-hour writes are priced differently from 5-minute writes, so usage.cache_creation_extended_input_tokens lets cost accounting tell them apart. See the Anthropic prompt caching guide for pricing and minimum cacheable lengths.
Inference Speed#
Set speed="fast" or speed="standard" to select the inference speed on supported models. The default, None, omits the parameter and leaves the API's default behavior unchanged. The option works with both streaming and non-streaming requests, including config.copy(speed="standard") overrides.
Some models require the fast-mode-2026-02-01 beta header for either explicit value, including "standard". AG2 does not add this model-specific header automatically; set it through default_headers when required:
Caller-supplied beta headers are preserved when AG2 adds its own tool-related betas. See Anthropic's fast-mode guide for current model availability, header requirements and pricing. Fast mode has premium pricing; enabling it does not make an unsupported model compatible.
Custom HTTP Client#
anthropic>=1 is built on httpx2, so http_client takes an httpx2.AsyncClient:
An httpx client is rejected
The anthropic SDK keeps no compatibility path for a legacy httpx.AsyncClient — it raises TypeError as soon as the client is built, where the OpenAI SDK would still accept one. Rebuild yours against httpx2; the two packages share an API, so this is usually just the import line.
Sampling Parameters#
temperature, top_p and top_k on AnthropicConfig are deprecated and are removed in 1.4.0. The Messages API version that anthropic 2.x sends (2026-10-21) no longer accepts them: a request carrying any of them is answered with HTTP 400 Extra inputs are not permitted, on every model, Claude Haiku 4.5 included. With anthropic 2.x, AG2 therefore drops the value; with anthropic 1.x it is still sent. Either way, config.create() emits a DeprecationWarning.
extra_body
Pass extra_body to forward provider-specific keys that have no dedicated field. AG2 does not interpret it, so the API still answers 400 to a sampling key written there.
Amazon Bedrock Authentication#
BedrockConfig authenticates in either of two ways, both resolved by the underlying AWS SDK:
1. AWS credentials (SigV4) — explicit keys, a profile_name, or the standard environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN). Recommended for production; credentials refresh automatically through aiobotocore.
2. Bedrock API keys (bearer token) — set the Amazon Bedrock API key as an environment variable and botocore uses bearer-token auth for Bedrock calls automatically (no other credentials needed):
A region is always required — pass region_name= or set AWS_DEFAULT_REGION. Notes on API keys:
- Short-term keys expire with the console session that minted them (max 12 hours) and are region-bound — generate the key in the same region you call.
- Long-term keys are backed by an auto-created IAM user; AWS recommends them for exploration only.
- API keys work only for Bedrock / Bedrock Runtime actions. If both a bearer token and AWS credentials are present, the bearer token wins for Bedrock calls.
Custom Session#
BedrockConfig is built on aiobotocore, so session takes an aiobotocore.session.AioSession — a boto3.Session is not accepted. Sharing one session across configs resolves credentials and loads the Bedrock service model once:
Google Vertex AI (Gemini)#
For Gemini on Vertex AI (Google Cloud), use the dedicated VertexAIConfig class. GeminiConfig covers the public Developer API (api_key); VertexAIConfig covers the Vertex path (GCP project, location, and Google-issued credentials).
Authentication accepts any of the following:
The service account needs the Vertex AI User (roles/aiplatform.user) IAM role on the project.
Use this path for impersonated credentials, workload identity, or any other google.auth.credentials.Credentials source.
Environment variables#
Instead of passing parameters explicitly, the underlying google-genai SDK resolves any field left unset from the following environment variables:
| Environment variable | Used by | Equivalent parameter | Notes |
|---|---|---|---|
GOOGLE_API_KEY | GeminiConfig | api_key | Takes precedence over GEMINI_API_KEY if both are set. |
GEMINI_API_KEY | GeminiConfig | api_key | Developer API key. |
GOOGLE_CLOUD_PROJECT | VertexAIConfig | project | GCP project ID. |
GOOGLE_CLOUD_LOCATION | VertexAIConfig | location | GCP region (or global). |
GOOGLE_APPLICATION_CREDENTIALS | VertexAIConfig | credentials | Path to a service-account JSON key file, read via ADC. |
With the three Vertex variables set in the environment, configuration collapses to just the model name:
export GOOGLE_CLOUD_PROJECT=my-gcp-project
export GOOGLE_CLOUD_LOCATION=us-central1
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account-key.json
Controlling Gemini Thinking#
Gemini 3 Pro models default to dynamic / unbounded thinking, which can cause individual calls to spend a large internal token budget before responding. Both GeminiConfig and VertexAIConfig accept thinking controls that map directly to Google's Thinking API.
Use thinking_level for Gemini 3 models, or thinking_budget for Gemini 2.5 models:
For full control (e.g. enabling include_thoughts), pass a google.genai.types.ThinkingConfig directly via thinking_config. When set, it takes precedence over the shorthand fields.
The number of thinking tokens consumed is reported on ModelResponse.usage.thinking_tokens and emitted as the gen_ai.usage.thinking_tokens OpenTelemetry attribute by TelemetryMiddleware.
Gemini Responses Cut at the Token Limit#
With long decode (Gemini 4 Argon, with max_output_tokens above the 64K per-request cap), Gemini stops a long response at that cap before the answer is complete, with the finish reason CONTINUATION and a token to resume from. From google-genai 2.30, the SDK resumes such a response automatically, request after request, until the model finishes. AG2 then returns the joined answer, and ModelResponse.usage is the sum of all those requests, since each one is billed on its own.
Set automatic_continuation=False on GeminiConfig or VertexAIConfig to turn the resumption off, for example to cap the cost of a single call:
A response cut short then reports response.finish_reason == "continuation", so it can be told apart from a finished one. With google-genai 2.29, which knows the finish reason but cannot resume, every cut response reports it this way.
Ollama Configuration#
OllamaConfig connects to a local or hosted Ollama server through the official ollama SDK (ollama>=0.5.2). Install the optional dependency with pip install "ag2[ollama]".
host defaults to http://localhost:11434; the SDK fills in the scheme and default port for a bare host such as gpu-box.
API Key#
Ollama Cloud needs a bearer token. Pass api_key and it is sent as Authorization: Bearer <key> on every request:
Custom HTTP Client#
Pass http_client to route requests through your own client, for a proxy, custom TLS, a transport or tracing. The ollama SDK is built on httpx, so http_client takes an httpx.AsyncClient:
host and api_key still apply on top of your client: AG2 supplies the URL and the bearer token for each request and leaves the client itself unmodified. Your client's own headers, timeout, proxy and transport are used as given.
You own the client's lifetime
AG2 never closes a client you pass in. Close it yourself with await client.aclose() when you are done with it.
httpx, not httpx2
Unlike OpenAICompletionsConfig and AnthropicConfig, which take an httpx2 client, Ollama's http_client is an httpx client. Use proxy= rather than proxies=, which current httpx rejects.
Tool Results#
When a tool round is sent back to the model, each result carries the name of the tool it answers as tool_name, as Ollama's tool-calling protocol describes. Results are still sent in call order: two parallel calls to the same tool are told apart by position only.
Z.AI (GLM) Configuration#
ZAIConfig connects to Z.AI's GLM models through the official zai-sdk. Install the optional dependency with pip install "ag2[zai]".
If api_key and base_url are not passed explicitly, the SDK resolves them from the ZAI_API_KEY and ZAI_BASE_URL environment variables; base_url otherwise defaults to the international endpoint (https://api.z.ai/api/paas/v4).
Controlling GLM Thinking#
GLM models accept reasoning controls that map directly to the Z.AI API. Set thinking to toggle the reasoning pass (True enables it, False disables it, unset uses the model default) and reasoning_effort to bound how much the model thinks:
Reasoning content is streamed as ModelReasoning events, and reasoning tokens are reported on ModelResponse.usage.thinking_tokens.
extra_body precedence
Pass extra_body to forward provider-specific keys that have no dedicated field. Explicit top-level fields always win over the same key inside extra_body — e.g. a top-level thinking=True overrides a thinking entry inside extra_body, while unrelated extra_body keys are merged through untouched.
xAI Configuration#
XAIConfig connects to xAI's Grok models through the official xai-sdk. Install the optional dependency with pip install "ag2[xai]".
If api_key is not passed explicitly, the SDK resolves it from the XAI_API_KEY environment variable.
Safety Identifier and Moderation Guidelines#
guidelines tightens moderation per category. Each entry declines one of nsfw, violence or profanity; a category you leave out keeps the model's default behavior.
safety_identifier tells xAI which end user is behind a request, so a policy violation is attributed to that user rather than to your whole API key. It must be stable and opaque: send a hash of your internal user ID, never an email address or display name. Because it differs per user, set it per request with config.copy(...). xAI prefers it over the older user field, which is still accepted.
An unknown category or behavior raises ValueError before the request is sent. The API rejects a request that lists a category twice. It also rejects guidelines on a multi-agent model, together with stop, or on a deferred completion.
Mistral Configuration#
MistralConfig connects to Mistral's chat-completions API through the official mistralai SDK. Install the optional dependency with pip install "ag2[mistral]".
If api_key is not passed explicitly, the SDK resolves it from the MISTRAL_API_KEY environment variable.
Point server_url at a different deployment (for example a private or regional endpoint) when you are not calling https://api.mistral.ai.
Reasoning Models#
Magistral and other reasoning models return their thinking trace alongside the answer. AG2 splits the two: the trace is emitted as ModelReasoning events, and only the answer text lands on ModelResponse.message.
Image Generation#
ImageGenerationTool is executed by Mistral, not locally. The call and its result are surfaced as BuiltinToolCallEvent and BuiltinToolResultEvent, and the generated image arrives as a UrlInput:
The image URL is a short-lived signed link, so download it promptly if you need to keep it. Mistral's tool takes no options, so size, quality, and output_format are ignored. The model may reply with the image and no commentary, leaving reply.body empty — the image is on the tool result either way.
Other builtin tools
Apart from image generation, Mistral's server-side tools (web_search, code_interpreter, document_library) belong to its Agents API and are rejected by chat-completions. Passing those AG2 builtin tools raises UnsupportedToolError rather than failing at request time.
OpenTelemetry version ceiling
The mistralai SDK pins opentelemetry-semantic-conventions<0.61, which transitively caps opentelemetry-api at 1.39.1. A clean install of ag2[mistral,tracing] resolves to a consistent OpenTelemetry 1.39.x stack and traces normally.
Adding ag2[mistral] to an environment that already has a newer OpenTelemetry downgrades opentelemetry-api on its own, leaving it mismatched against the newer SDK. Reinstall the pair together, or hold the newer versions explicitly — the SDK itself works fine with current OpenTelemetry, the pin is just over-tight:
OpenAI Decisions Configuration#
OpenAIDecisionsConfig connects to OpenAI's Decisions API (POST /v1/decisions), which is in public beta. It ships with the openai extra (openai>=3.27.0), and like the other OpenAI configs reads OPENAI_API_KEY when api_key is not passed. model defaults to gpt-6-luna, the only model the endpoint serves today.
Like TypeSafe's Jev, the Decisions API does not generate text. It answers typed questions about the conversation: the probability that a condition is true, one option out of several, or a score on an ordered rubric, along with the probabilities behind it. Use it to classify, route or grade cheaply, then hand the text work to a generative agent.
The response schema is the question#
An agent on OpenAIDecisionsConfig must set response_schema, and the schema picks the question type:
response_schema | Question type | await reply.content() returns |
|---|---|---|
bool | predicate | True once the probability reaches boolean_threshold (default 0.5) |
a number schema bounded to 0..1, e.g. ResponseSchema.from_schema({"type": "number", "minimum": 0, "maximum": 1}, name="churn", description="How likely is churn?") | predicate | the probability as JSON text, unparsed; reply.response.metadata["probability"] has it as a float |
an Enum of strings | choice | the selected member |
an IntEnum numbered 0..n-1 (at least two levels) | score | the member nearest the returned score |
a flat class (dataclass, TypedDict, BaseModel) whose required fields are each one of the above | one question per field, in one request | an instance of the class; metadata is {field: answer} |
Anything else (str, int, a nested or optional field, a list, a PromptedSchema) raises UnsupportedResponseSchemaError before a request is sent. See Several questions at once for the class form.
The agent prompt and the schema's description (an explicit ResponseSchema(..., description=...), an Annotated[..., Question("...")] marker, or an Enum's class docstring, in that order of precedence) are joined into the question's instructions. Every question type requires instructions, so a schema with no description on an agent with no prompt raises ValueError locally.
Describing options and levels#
The string literal under each Enum member becomes that choice's description. For an IntEnum, member names become the level labels (WORKAROUND_AVAILABLE is sent as Workaround available), and the docstrings become level descriptions. Unlike Jev, a level without a docstring is allowed.
descriptions on the config overrides or supplies descriptions without touching the type, keyed by choice value or by score level as a string ({"0": "...", "1": "..."}). descriptions outrank Annotated[..., Question(options=...)], which outranks member docstrings (see Decision Models). Member docstrings are read from the source file, so use either for enums defined in a REPL or notebook.
Images#
Image inputs are sent inline as base64 data URLs, the only image form the endpoint accepts. ImageInput(path=...), ImageInput(data=..., media_type=...) and ImageInput("data:image/png;base64,...") all work, and vendor_metadata={"detail": ...} is forwarded. Hosted image URLs, file_ids and non-image files raise UnsupportedInputError. There is no minimum size, and the cap is on area rather than on a side: a 5500×5500 image passes, while 5600×5600 is answered with 400 ... exceeds the limit of 30000 patches.
Images that every question should carry, such as a reference photo, go in question_instructions on the config: a string, or a list of strings, inputs and OpenAI parts (input_text / input_image / input_json). They follow the agent prompt in the question's instructions, and a hosted https image URL is allowed there. See Decision Models for an example.
Refusals and what the Decisions API does not do#
- Refusals. The API may decline a question. That raises
DecisionRefusedError(fromag2.config.openai) rather than returning a made-up answer. With a class schema, a refusal of any one field fails the whole reply, and the error'snameis that field. - Tools. Passing any tool raises
UnsupportedToolError. - Streaming. There is no
streamingfield; the answer arrives as oneModelMessage. - Files API.
create_files_client()raisesNotImplementedError. - History roles. The endpoint takes only user messages, so earlier assistant replies and tool results are sent as user text tagged
[Assistant]/[Tool result]. - Dependent questions. A class schema batches independent questions into one request; questions that depend on an earlier answer still need separate
ask()calls.
safety_identifier, extra_body and the usual client options (base_url, organization, project, timeout, max_retries, default_headers, http_client) are forwarded to the OpenAI SDK. Usage lands on ModelResponse.usage. Decisions bill input tokens only.
For recipes covering classification, routing, detection, scoring, ranking, verification, feature extraction and image inspection, see Decision Models.
TypeSafe (Jev) Configuration#
TypeSafeConfig connects to TypeSafe AI's Jev decision model through the official typesafe-sdk. Install the optional dependency with pip install "ag2[typesafe]".
For task recipes with Jev, see Decision Models.
Jev is not a text generator. It answers typed questions about the conversation so far — yes or no, one label out of several, a level on a rubric, or where something is in an image — and returns the answer together with its confidence and the probability of every option. That makes it a cheap, predictable front for a generative agent: route a ticket, gate a reply, grade a draft, then hand the text work to another provider.
If api_key and base_url are not passed explicitly, the SDK resolves them from the TYPESAFE_API_KEY and TYPESAFE_BASE_URL environment variables. model defaults to jev-latest.
The response schema is the question#
There is no free-text path, so an agent on TypeSafeConfig must set response_schema, and the schema picks which of Jev's primitives is asked:
response_schema | Jev asks | await reply.content() returns |
|---|---|---|
bool | a yes/no question | True once the yes-probability reaches boolean_threshold (default 0.5) |
a number schema bounded to 0..1, e.g. ResponseSchema.from_schema({"type": "number", "minimum": 0, "maximum": 1}, name="churn", description="How likely is churn?") | a yes/no question | the yes-probability as JSON text, unparsed (from_schema schemas are not validated); reply.response.metadata["noul"] has it as a float |
an Enum of strings | a choice between its members | the selected member |
an IntEnum numbered 0..n-1 with 2–10 levels | a score on that rubric | the member nearest Jev's expected score |
BoundingBox | where something is in an image (see Locating things in an image) | the BoundingBox: each edge as a 0..1 fraction of the image |
a flat class (dataclass, TypedDict, BaseModel) whose required fields are each a bool, 0..1 float, Enum or IntEnum | one question per field, in one request | an instance of the class; metadata is {field: answer} |
Anything else — str, int, a nested or optional field, a list, a PromptedSchema — raises UnsupportedResponseSchemaError before any request is sent. See Several questions at once for the class form; criteria on the config applies to single-question schemas only.
The agent prompt frames the question; the schema's own description asks it. That is an explicit ResponseSchema(..., description=...), an Annotated[..., Question("...")] marker, or an Enum's class docstring, in that order of precedence. The prompt and the question are joined, so the router above asks "You triage customer support tickets. Which team should handle this ticket?".
A yes/no question has to be asked
response_schema=bool with no prompt, no description and no criteria raises ValueError locally. The API rejects a bare yes/no question, and failing before the request keeps the error actionable.
Locating things in an image#
BoundingBox asks Jev where something is in an image. The agent prompt says what to find, the image goes in with the question, and the answer is the box around it.
Images and box questions are experimental on TypeSafe's side, so two settings are required: the jev-preview model and the opt-in header TypeSafe-Danger-Zone-Images: true, sent through headers. Without the header the API answers 400 Images are experimental. Each side of an image must be between 280 and 1568 pixels: a smaller or larger one, such as a 256×256 icon or a 1600×1200 photo, is answered with 400 Invalid image dimensions, so resize it before sending.
The edges are fractions of the image, measured from its top-left corner: top and bottom of its height, left and right of its width. Multiply them by the image size to get pixels, as in the last line above.
Send the image as ImageInput(path=...) or ImageInput(data=..., media_type=...). A URL, an uploaded file_id and non-image media raise UnsupportedInputError.
A box needs an image
response_schema=ResponseSchema(BoundingBox) with no image in the request raises ValueError locally. Without the check the API would not fail; it would answer with an invented box and a confidence near zero.
Treat the box as an estimate
jev-preview finds the right region, but its edges can be off by 0.1–0.2 of the image, and confidence is often low (around 0.3 in our checks). Read reply.response.metadata["confidence"] before acting on a box, and leave a margin when you crop or draw it.
Describing the options#
The string literal under each Enum / IntEnum member describes that option to Jev. Python discards those strings at runtime, so AG2 reads them back from the class source.
A score rubric requires a description for every level; a choice works without them. criteria on the config overrides or supplies descriptions without touching the type (Annotated[..., Question(options=...)] does the same beside the type, and criteria wins) — keyed by choice label, by score level as a string, or by "true" / "false" for a yes/no question:
Member docstrings need the source file
Docstrings are read with inspect.getsource, so an Enum defined in a REPL, a notebook cell or a frozen app has none. Pass criteria or Annotated[..., Question(options=...)] there; criteria wins when both are set.
Confidence and probabilities#
The SDK answer, minus its type, is kept on reply.response.metadata. A yes/no answer is just noul, the yes-probability. A choice carries choice, confidence and probabilities per label. A score carries score (the expected value on the rubric, before snapping), confidence, the rubric legend and probabilities per level.
What Jev does not do#
- Tools. Passing any tool raises
UnsupportedToolError; keep tools on the generative agent Jev routes to. - Streaming. Requests are non-streaming; there is no
streamingfield, and the answer arrives as oneModelMessage. - Files and media.
create_files_client()raisesNotImplementedError, and only text, structuredDataInputand image parts (experimental, see Locating things in an image) are sent — anything else raisesUnsupportedInputError. - Free text.
reply.bodyis the answer as JSON, not prose.
retry takes a typesafe_sdk.RetryPolicy and http_client an httpx2.AsyncClient; both are forwarded to the SDK untouched. Token counts land on ModelResponse.usage as prompt_tokens / completion_tokens when the API reports them.
Structured data extraction, one of the decision shapes in TypeSafe's docs, is the one this provider cannot express: an object schema is rejected, since Jev answers one decision per question.
Self-Hosted and OpenAI-Compatible Models (vLLM, LM Studio, etc.)#
If you are using a self-hosted model or an API that is compatible with the OpenAI format (such as vLLM, LM Studio, FastChat, or Together AI), you can use the OpenAICompletionsConfig class and specify a custom base_url.
Tip
If you are running a self-hosted server via HTTPS without a valid SSL certificate (e.g., a local self-signed certificate), you can disable SSL checks by passing a custom httpx2.AsyncClient with verify=False to the configuration:
http_client is an httpx2 client
From openai>=3 the OpenAI SDK uses httpx2, which ag2[openai] installs for you. OpenAICompletionsConfig and OpenAIResponsesConfig annotate http_client accordingly, and ag2 forwards whatever you pass to the SDK untouched.
A legacy httpx.AsyncClient still works at runtime: the SDK keeps a compatibility path for it, though it fails static type checking and the SDK's migration guide calls that path a migration aid that may be discontinued. Build the client with httpx2.AsyncClient to be done with it.
Each provider takes the client its own SDK takes, so the annotation differs by provider — AnthropicConfig.http_client, for instance, follows whichever package the pinned anthropic release is built on.
TLS certificates come from the operating system
httpx2 verifies certificates against the operating system trust store, not a bundled one. Nothing extra is needed on a normal machine or a standard base image, but a minimal container without system CA certificates will fail to connect. Install the distribution's CA bundle (e.g. ca-certificates), or point the client at one explicitly:
import httpx2
from ag2.config import OpenAICompletionsConfig
config = OpenAICompletionsConfig(model="gpt-4o", http_client=httpx2.AsyncClient(verify="/path/to/ca-bundle.crt"))
certifi remains installed — ag2's own core depends on httpx — but the OpenAI client no longer reads it. Its presence is not evidence that TLS is configured there.
Extra Body Parameters#
Some OpenAI API-compatible providers require additional, provider-specific parameters in the request body. Use the extra_body parameter on OpenAICompletionsConfig to pass these through directly to the API call.
This is useful for enabling features like extended thinking on self-hosted or third-party models:
Reusing and Overriding Configurations#
Model configurations are immutable. If you need to reuse a configuration for multiple agents with slight variations (e.g., changing the model version or adjusting the temperature), use the .copy() method. This creates a new updated instance without mutating the original configuration.
Delaying Model Configuration#
In many use cases, you may want to separate the logic of defining your agent (tools, system messages, instructions) from configuring the specific model it uses. This allows you to construct an agent once and dynamically provide the model configuration later during execution.
You can accomplish this by passing the configuration to the .ask() method when interacting with the agent. This is especially useful for applications like web servers where the user might bring their own API key or choose a different model on the fly.
Warning
Providing a configuration or client directly to the ask() method completely overrides the original model configuration assigned to the agent for that specific turn.