autogen.agentchat.realtime.experimental.clients.oai.OpenAIRealtimeWebRTCClient
OpenAIRealtimeWebRTCClient
(Experimental) Client for OpenAI Realtime API that uses WebRTC protocol.
(Experimental) Client for OpenAI Realtime API.
Parameters:Name | Description |
---|---|
llm_config | The config for the client. Type: dict[str, typing.Any] |
websocket | Type: WebSocket |
logger | Type: logging.Logger | None Default: None |
Static Methods
get_factory
Create a Realtime API client.
Parameters:Name | Description |
---|---|
llm_config | Type: dict[str, typing.Any] |
logger | Type: logging.Logger |
**kwargs | Type: Any |
Type | Description |
---|---|
Callable[[], autogen.agentchat.realtime.experimental.clients.realtime_client.RealtimeClientProtocol] | None | RealtimeClientProtocol: The Realtime API client is returned if the model matches the pattern |
Instance Attributes
logger
Get the logger for the OpenAI Realtime API.
Instance Methods
connect
Connect to the OpenAI Realtime API.
In the case of WebRTC, we pass connection information over the websocket, so that javascript on the other end of websocket open actual connection to OpenAI
read_events
Read messages from the OpenAI Realtime API. Again, in case of WebRTC, we do not read OpenAI messages directly since we do not hold connection to OpenAI. Instead we read messages from the websocket, and javascript client on the other side of the websocket that is connected to OpenAI is relaying events to us.
send_audio
Send audio to the OpenAI Realtime API.
Parameters:Name | Description |
---|---|
audio | The audio to send. Type: str |
send_function_result
Send the result of a function call to the OpenAI Realtime API.
Parameters:Name | Description |
---|---|
call_id | The ID of the function call. Type: str |
result | The result of the function call. Type: str |
send_text
Send a text message to the OpenAI Realtime API.
Parameters:Name | Description |
---|---|
role | The role of the message. Type: Literal['user', 'assistant', 'system'] |
text | The text of the message. Type: str |
session_init_data
Control initial session with OpenAI.
session_update
Send a session update to the OpenAI Realtime API.
In the case of WebRTC we can not send it directly, but we can send it to the javascript over the websocket, and rely on it to send session update to OpenAI
Parameters:Name | Description |
---|---|
session_options | The session options to update. Type: dict[str, typing.Any] |
truncate_audio
Truncate audio in the OpenAI Realtime API.
Parameters:Name | Description |
---|---|
audio_end_ms | The end of the audio to truncate. Type: int |
content_index | The index of the content to truncate. Type: int |
item_id | The ID of the item to truncate. Type: str |