oai.client
ModelClient
A client class must implement the following methods:
- create must return a response object that implements the ModelClientResponseProtocol
- cost must return the cost of the response
- get_usage must return a dict with the following keys:
- prompt_tokens
- completion_tokens
- total_tokens
- cost
- model
This class is used to create a client that can be used by OpenAIWrapper. The response returned from create must adhere to the ModelClientResponseProtocol but can be extended however needed. The message_retrieval method must be implemented to return a list of str or a list of messages from the response.
message_retrieval
Retrieve and return a list of strings or a list of Choice.Message from the response.
NOTE: if a list of Choice.Message is returned, it currently needs to contain the fields of OpenAI’s ChatCompletion Message object, since that is expected for function or tool calling in the rest of the codebase at the moment, unless a custom agent is being used.
get_usage
Return usage summary of the response using RESPONSE_USAGE_KEYS.
OpenAIClient
Follows the Client protocol and wraps the OpenAI client.
message_retrieval
Retrieve the messages from the response.
create
Create a completion for a given config using openai’s client.
Arguments:
client
- The openai client.params
- The params for the completion.
Returns:
The completion.
cost
Calculate the cost of the response.
OpenAIWrapper
A wrapper class for openai client.
__init__
Arguments:
config_list
- a list of config dicts to override the base_config. They can contain additional kwargs as allowed in the create method. E.g.,
base_config
- base config. It can contain both keyword arguments for openai client and additional kwargs. When using OpenAI or Azure OpenAI endpoints, please specify a non-empty ‘model’ either inbase_config
or in each config ofconfig_list
.
register_model_client
Register a model client.
Arguments:
model_client_cls
- A custom client class that follows the ModelClient interface**kwargs
- The kwargs for the custom client class to be initialized with
create
Make a completion for a given config using available clients. Besides the kwargs allowed in openai’s [or other] client, we allow the following additional kwargs. The config in each client will be overridden by the config.
Arguments:
- context (Dict | None): The context to instantiate the prompt or messages. Default to None.
It needs to contain keys that are used by the prompt template or the filter function.
E.g.,
prompt="Complete the following sentence: \{prefix}, context=\{"prefix": "Today I feel"}
. The actual prompt will be: “Complete the following sentence: Today I feel”. More examples can be found at templating. - cache (AbstractCache | None): A Cache object to use for response cache. Default to None. Note that the cache argument overrides the legacy cache_seed argument: if this argument is provided, then the cache_seed argument is ignored. If this argument is not provided or None, then the cache_seed argument is used.
- agent (AbstractAgent | None): The object responsible for creating a completion if an agent.
- (Legacy) cache_seed (int | None) for using the DiskCache. Default to 41. An integer cache_seed is useful when implementing “controlled randomness” for the completion. None for no caching.
Note
- this is a legacy argument. It is only used when the cache argument is not provided.- filter_func (Callable | None): A function that takes in the context and the response and returns a boolean to indicate whether the response is valid. E.g.,
- allow_format_str_template (bool | None): Whether to allow format string template in the config. Default to false.
- api_version (str | None): The api version. Default to None. E.g., “2024-02-01”.
Raises:
- RuntimeError: If all declared custom model clients are not registered
- APIError: If any model client create call raises an APIError
print_usage_summary
Print the usage summary.
clear_usage_summary
Clear the usage summary.
extract_text_or_completion_object
Extract the text or ChatCompletion objects from a completion or chat response.
Arguments:
response
ChatCompletion | Completion - The response from openai.
Returns:
A list of text, or a list of ChatCompletion objects if function_call/tool_calls are present.