Usage tracking with AutoGen
1. Use AutoGen’s OpenAIWrapper for cost estimation
The OpenAIWrapper
from autogen
tracks token counts and costs of your
API calls. Use the create()
method to initiate requests and
print_usage_summary()
to retrieve a detailed usage report, including
total cost and token usage for both cached and actual requests.
mode=["actual", "total"]
(default): print usage summary for non-caching completions and all completions (including cache).mode='actual'
: only print non-cached usage.mode='total'
: only print all usage (including cache).
Reset your session’s usage data with clear_usage_summary()
when
needed.
2. Track cost and token count for agents
We also support cost estimation for agents. Use
Agent.print_usage_summary()
to print the cost summary for the agent.
You can retrieve usage summary in a dict using
Agent.get_actual_usage()
and Agent.get_total_usage()
. Note that
Agent.reset()
will also reset the usage summary.
To gather usage data for a list of agents, we provide an utility
function autogen.gather_usage_summary(agents)
where you pass in a list
of agents and gather the usage summary.
3. Custom token price for up-to-date cost estimation
AutoGen tries to keep the token prices up-to-date. However, you can pass
in a price
field in config_list
if the token price is not listed or
up-to-date. Please creating an issue or pull request to help us keep the
token prices up-to-date!
Note: in json files, the price should be a list of two floats.
Example Usage:
Caution when using Azure OpenAI!
If you are using azure OpenAI, the model returned from completion doesn’t have the version information. The returned model is either ‘gpt-35-turbo’ or ‘gpt-4’. From there, we are calculating the cost based on gpt-3.5-turbo-0125: (0.0005, 0.0015) per 1k prompt and completion tokens and gpt-4-0613: (0.03, 0.06). This means the cost can be wrong if you are using a different version from azure OpenAI.
This will be improved in the future. However, the token count summary is accurate. You can use the token count to calculate the cost yourself.
Requirements
AutoGen requires Python>=3.9
:
Set your API Endpoint
The
config_list_from_json
function loads a list of configurations from an environment variable or
a json file.
It first looks for environment variable “OAI_CONFIG_LIST” which needs to be a valid json string. If that variable is not found, it then looks for a json file named “OAI_CONFIG_LIST”. It filters the configs by tags (you can filter by other keys as well).
The config list looks like the following:
You can set the value of config_list in any way you prefer. Please refer to this notebook for full code examples of the different methods.
OpenAIWrapper with cost estimation
OpenAIWrapper with custom token price
Usage Summary for OpenAIWrapper
When creating a instance of OpenAIWrapper, cost of all completions from
the same instance is recorded. You can call print_usage_summary()
to
checkout your usage summary. To clear up, use clear_usage_summary()
.
Usage Summary for Agents
Agent.print_usage_summary()
will print the cost summary for the agent.Agent.get_actual_usage()
andAgent.get_total_usage()
will return the usage summary in a dict. When an agent doesn’t use LLM, they will return None.Agent.reset()
will reset the usage summary.autogen.gather_usage_summary
will gather the usage summary for a list of agents.