Skip to content

Multimodal Inputs#

AG2 agents can process images, audio, video, and documents alongside text. The input event system provides a unified API across providers — you create inputs the same way regardless of which model you use.

Input Types#

Factory Function Creates Description
ImageInput(...) Image input JPEG, PNG, GIF, WebP
AudioInput(...) Audio input WAV, MP3, OGG, FLAC, AAC
VideoInput(...) Video input MP4, WebM, MOV, MKV, MPEG
DocumentInput(...) Document input PDF, TXT, HTML, Markdown, CSV, JSON, Office formats

Each factory function supports multiple ways to provide the data:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from ag2.events import ImageInput, AudioInput, VideoInput, DocumentInput

# From a URL
image = ImageInput("https://example.com/photo.jpg")

# From a local file path
image = ImageInput(path="photo.jpg")

# From raw bytes
image = ImageInput(data=raw_bytes, media_type="image/png")

# From a pre-uploaded file ID (provider-specific)
image = ImageInput(file_id="file-abc123")

Using Inputs with Agents#

Pass inputs directly to agent.ask() as positional arguments alongside text:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from ag2 import Agent
from ag2.config import GeminiConfig
from ag2.events import ImageInput

agent = Agent(
    "vision_agent",
    "You are a helpful assistant that describes images.",
    config=GeminiConfig(model="gemini-3-flash-preview"),
)

image = ImageInput("https://example.com/photo.jpg")
reply = await agent.ask("Describe this image in detail.", image)
print(reply.body)

You can pass multiple inputs in a single request:

1
2
3
4
image1 = ImageInput("https://example.com/before.jpg")
image2 = ImageInput("https://example.com/after.jpg")

reply = await agent.ask("Compare these two images.", image1, image2)

Provider Support#

Not all providers support all input types. The table below shows what each provider accepts:

Input Type OpenAI OpenAI Responses Gemini Anthropic xAI Mistral Bedrock
Text Yes Yes Yes Yes Yes Yes Yes
Image (URL) Yes Yes Yes Yes Yes Yes -
Image (binary) Yes Yes Yes Yes Yes Yes Yes
Audio (URL) - - Yes - - - -
Audio (binary) Yes - Yes - - - -
Video (URL) - - Yes - - - -
Video (binary) - - Yes - - - Yes
Document (URL) - Yes Yes Yes Yes Yes -
Document (binary) - - Yes Yes Yes Yes Yes
File ID - Yes - Yes Yes Yes -

If you pass an unsupported input type to a provider, an UnsupportedInputError is raised with a clear message indicating what is not supported and by which provider.

Provider-Specific Details#

Gemini#

Gemini has the broadest multimodal support — it accepts images, audio, video, and documents in all forms (URL, binary, and local file path).

YouTube URLs are supported directly:

1
2
3
4
from ag2.events import VideoInput

video = VideoInput("https://www.youtube.com/watch?v=dQw4w9WgXcQ")
reply = await agent.ask("Summarize this video.", video)

Google Files API — for large files (>20MB), upload via the Google Files API first and pass the returned URI:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
from google import genai
from ag2.events import VideoInput

client = genai.Client()
uploaded = client.files.upload(file="large_video.mp4")

# Wait for processing to complete
import time
while uploaded.state.name == "PROCESSING":
    time.sleep(2)
    uploaded = client.files.get(name=uploaded.name)

video = VideoInput(uploaded.uri)
reply = await agent.ask("Describe this video.", video)

Vendor Metadata#

Gemini supports provider-specific settings via vendor_metadata on binary inputs. These map to Gemini Part fields:

Key Type Description
media_resolution str | dict[str, Any] Controls token allocation per image/video frame
video_metadata dict[str, Any] Video clipping (start_offset, end_offset) and frame rate (fps)
display_name str Display name for the file

Media resolution — control quality vs cost tradeoff for images and video frames:

1
2
3
4
5
6
7
8
from ag2.events import ImageInput

# Lower resolution = fewer tokens = lower cost
image = ImageInput(
    data=raw_bytes,
    media_type="image/jpeg",
    vendor_metadata={"media_resolution": "MEDIA_RESOLUTION_LOW"},
)

Available values: MEDIA_RESOLUTION_LOW, MEDIA_RESOLUTION_MEDIUM, MEDIA_RESOLUTION_HIGH, MEDIA_RESOLUTION_ULTRA_HIGH.

A dict is also accepted and maps straight onto Gemini's PartMediaResolution, whose level and num_tokens keys are mutually exclusive — set exactly one:

1
2
3
4
5
6
7
from ag2.events import ImageInput

image = ImageInput(
    data=raw_bytes,
    media_type="image/jpeg",
    vendor_metadata={"media_resolution": {"level": "MEDIA_RESOLUTION_MEDIUM"}},
)

Setting both keys is rejected

Passing {"level": ..., "num_tokens": ...} together makes Gemini reject the request with 400 INVALID_ARGUMENT (oneof field 'value' is already set).

Video clipping and frame rate — process only a portion of a video or adjust the sampling rate:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from ag2.events import VideoInput

video = VideoInput(
    path="lecture.mp4",
    vendor_metadata={
        "video_metadata": {
            "start_offset": "60s",
            "end_offset": "120s",
            "fps": 0.5,
        },
    },
)
reply = await agent.ask("Summarize this section of the video.", video)

Display name — attach a name to the file for reference:

1
2
3
4
5
6
from ag2.events import DocumentInput

doc = DocumentInput(
    path="report.pdf",
    vendor_metadata={"display_name": "Q4 Financial Report"},
)

OpenAI#

OpenAI supports images via both the Completions and Responses APIs. Audio binary input (WAV, MP3) is supported in the Completions API. The Responses API additionally supports file IDs and document URLs.

Vendor Metadata#

OpenAI supports vendor_metadata for image detail control:

1
2
3
4
5
6
7
from ag2.events import ImageInput

image = ImageInput(
    data=raw_bytes,
    media_type="image/png",
    vendor_metadata={"detail": "low"},  # "low", "high", or "auto"
)

Anthropic#

Anthropic supports images (JPEG, PNG, GIF, WebP) and documents (PDF) via URL, base64, or File ID. Audio and video are not supported.

File ID — upload files via the Anthropic Files API (beta) and reference by ID:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
import anthropic
from ag2.events import ImageInput, DocumentInput

client = anthropic.Anthropic()

# Upload an image
uploaded = client.beta.files.upload(
    file=("photo.jpg", open("photo.jpg", "rb"), "image/jpeg"),
)

# Reference by file_id — filename determines block type (image vs document)
image = ImageInput(file_id=uploaded.id, filename="photo.jpg")
reply = await agent.ask("Describe this image.", image)

Vendor Metadata#

Anthropic supports vendor_metadata for prompt caching on content blocks:

1
2
3
4
5
6
from ag2.events import DocumentInput

doc = DocumentInput(
    path="report.pdf",
    vendor_metadata={"cache_control": {"type": "ephemeral"}},
)

xAI#

xAI supports images (URL and binary), documents (URL and binary), and pre-uploaded file IDs. Audio and video are not currently supported — passing them raises UnsupportedInputError.

File ID — reference a file previously uploaded via the xAI Files API:

1
2
3
4
from ag2.events import ImageInput, DocumentInput

image = ImageInput(file_id="file-abc123", filename="photo.jpg")
doc = DocumentInput(file_id="file-xyz789", filename="report.pdf")

Vendor Metadata#

xAI reads detail for image quality control from two different attributes depending on the input source — vendor_metadata for binary, metadata for URL. Mixing them up means the value is silently ignored and xAI falls back to "auto".

Binary image — set detail via vendor_metadata:

1
2
3
4
5
6
7
8
from ag2.events import BinaryInput, BinaryType

image = BinaryInput(
    raw_bytes,
    media_type="image/png",
    kind=BinaryType.IMAGE,
    vendor_metadata={"detail": "low"},  # "low", "high", or "auto"
)

URL image — set detail via metadata (not vendor_metadata):

1
2
3
4
5
6
7
from ag2.events import UrlInput, BinaryType

image = UrlInput(
    "https://example.com/photo.jpg",
    kind=BinaryType.IMAGE,
    metadata={"detail": "low"},
)

Note

The factory ImageInput(url=...) does not forward metadata. To configure detail on a URL image, construct UrlInput directly as shown above.

Document filename — xAI requires a filename for binary documents. When sending raw bytes, either provide one via vendor_metadata={"filename": ...}, or rely on the auto-derived fallback (file.<subtype> from the media type, e.g. file.pdf for application/pdf):

1
2
3
4
5
6
7
8
from ag2.events import BinaryInput, BinaryType

doc = BinaryInput(
    pdf_bytes,
    media_type="application/pdf",
    kind=BinaryType.DOCUMENT,
    vendor_metadata={"filename": "Q4-report.pdf"},
)

Mistral#

Mistral accepts images and documents by URL, binary, or file ID. Audio and video raise UnsupportedInputError. Plain text counts as a document, so DocumentInput(data=b"...", media_type="text/plain") works alongside PDFs.

Files API — uploads must use the ocr purpose to be referenceable from a message. That is the default in AG2, so a plain upload is ready to use:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
from ag2.config import MistralConfig
from ag2.events import FileIdInput
from ag2.files import FilesAPI

config = MistralConfig(model="mistral-small-latest")
files = FilesAPI(config)
uploaded = await files.upload(data=pdf_bytes, filename="report.pdf")

# Reference it as many times as you like — the bytes are never resent
reply = await agent.ask("Summarise this report.", FileIdInput(file_id=uploaded.file_id))

The other purposes are content-validated: batch and fine-tune require newline-delimited JSON in a specific schema and reject anything else with HTTP 422.

Remote URLs are fetched by Mistral

URL inputs are retrieved server-side, so the host must serve them to Mistral. Some hosts refuse, and the API returns "File could not be fetched from url". Pass the bytes instead when that happens.

Vendor Metadata#

Mistral reads detail for image quality from vendor_metadata for binary images and metadata for URL images — the same split as xAI:

1
2
3
4
5
6
7
8
from ag2.events import BinaryInput, BinaryType

image = BinaryInput(
    raw_bytes,
    media_type="image/png",
    kind=BinaryType.IMAGE,
    vendor_metadata={"detail": "low"},  # "low", "high", or "auto"
)

Document name — taken from metadata={"filename": ...} for URL documents and vendor_metadata={"filename": ...} for binary ones (set automatically when using path=).

Amazon Bedrock#

The Bedrock Converse API accepts binary sources only — images (JPEG, PNG, GIF, WebP), documents (PDF, CSV, DOC, DOCX, XLS, XLSX, HTML, TXT, Markdown), and video (MP4, WebM, MOV, MKV, and more; Amazon Nova models). URL inputs and file IDs raise UnsupportedInputError — Bedrock has no Files API, so source data from a URL must be downloaded and passed as bytes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
from ag2 import Agent
from ag2.config import BedrockConfig
from ag2.events import DocumentInput, ImageInput

agent = Agent(
    "vision_agent",
    "You describe images and summarize documents.",
    config=BedrockConfig(model="us.amazon.nova-lite-v1:0", region_name="us-east-1"),
)

image = ImageInput(path="photo.jpg")
doc = DocumentInput(data=pdf_bytes, media_type="application/pdf")
reply = await agent.ask("Describe the image and summarize the document.", image, doc)

Note

Modality support also depends on the model behind the Converse API: Amazon Nova models accept images, documents, and video; many others (e.g. DeepSeek) are text-only and return a ValidationException from AWS for non-text blocks. The provider raises UnsupportedInputError only for inputs the Converse API itself cannot carry.

Document name — Converse requires a name for document blocks. It is taken from vendor_metadata={"filename": ...} (set automatically when using path=), sanitized to the characters Converse allows (alphanumerics, single spaces, hyphens, parentheses, brackets), and falls back to "document" when absent:

1
2
3
4
5
6
7
8
from ag2.events import BinaryInput, BinaryType

doc = BinaryInput(
    pdf_bytes,
    media_type="application/pdf",
    kind=BinaryType.DOCUMENT,
    vendor_metadata={"filename": "Q4 report.pdf"},
)