Runtime Logging with AG2#

AG2 offers utilities to log data for debugging and performance analysis. This notebook demonstrates how to use them.

we log data in different modes: - SQlite Database - File

In general, users can initiate logging by calling autogen.runtime_logging.start() and stop logging by calling autogen.runtime_logging.stop()

import json

import pandas as pd

import autogen
from autogen import AssistantAgent, LLMConfig, UserProxyAgent

# Setup API key. Add your own API key to config file or environment variable
llm_config = LLMConfig.from_json(path="OAI_CONFIG_LIST", temperature=0.9)

# Start logging
logging_session_id = autogen.runtime_logging.start(config={"dbname": "logs.db"})
print("Logging session ID: " + str(logging_session_id))

# Create an agent workflow and run it
assistant = AssistantAgent(name="assistant", llm_config=llm_config)
user_proxy = UserProxyAgent(
    name="user_proxy",
    code_execution_config=False,
    human_input_mode="NEVER",
    is_termination_msg=lambda msg: "TERMINATE" in msg["content"],
)

user_proxy.initiate_chat(
    assistant, message="What is the height of the Eiffel Tower? Only respond with the answer and terminate"
)
autogen.runtime_logging.stop()

Getting Data from the SQLite Database#

logs.db should be generated, by default it’s using SQLite database. You can view the data with GUI tool like sqlitebrowser, using SQLite command line shell or using python script:

def get_log(dbname="logs.db", table="chat_completions"):
    import sqlite3

    con = sqlite3.connect(dbname)
    query = f"SELECT * from {table}"
    cursor = con.execute(query)
    rows = cursor.fetchall()
    column_names = [description[0] for description in cursor.description]
    data = [dict(zip(column_names, row)) for row in rows]
    con.close()
    return data

def str_to_dict(s):
    return json.loads(s)

log_data = get_log()
log_data_df = pd.DataFrame(log_data)

log_data_df["total_tokens"] = log_data_df.apply(
    lambda row: str_to_dict(row["response"])["usage"]["total_tokens"], axis=1
)

log_data_df["request"] = log_data_df.apply(lambda row: str_to_dict(row["request"])["messages"][0]["content"], axis=1)

log_data_df["response"] = log_data_df.apply(
    lambda row: str_to_dict(row["response"])["choices"][0]["message"]["content"], axis=1
)

log_data_df

Computing Cost#

One use case of logging data is to compute the cost of a session.

# Sum totoal tokens for all sessions
total_tokens = log_data_df["total_tokens"].sum()

# Sum total cost for all sessions
total_cost = log_data_df["cost"].sum()

# Total tokens for specific session
session_tokens = log_data_df[log_data_df["session_id"] == logging_session_id]["total_tokens"].sum()
session_cost = log_data_df[log_data_df["session_id"] == logging_session_id]["cost"].sum()

print("Total tokens for all sessions: " + str(total_tokens) + ", total cost: " + str(round(total_cost, 4)))
print(
    "Total tokens for session "
    + str(logging_session_id)
    + ": "
    + str(session_tokens)
    + ", cost: "
    + str(round(session_cost, 4))
)

Log data in File mode#

By default, the log type is set to sqlite as shown above, but we introduced a new parameter for the autogen.runtime_logging.start()

the logger_type = "file" will start to log data in the File mode.

import pandas as pd

import autogen
from autogen import AssistantAgent, LLMConfig, UserProxyAgent

# Setup API key. Add your own API key to config file or environment variable
llm_config = LLMConfig.from_json(path="OAI_CONFIG_LIST", temperature=0.9)

# Start logging with logger_type and the filename to log to
logging_session_id = autogen.runtime_logging.start(logger_type="file", config={"filename": "runtime.log"})
print("Logging session ID: " + str(logging_session_id))

# Create an agent workflow and run it
assistant = AssistantAgent(name="assistant", llm_config=llm_config)
user_proxy = UserProxyAgent(
    name="user_proxy",
    code_execution_config=False,
    human_input_mode="NEVER",
    is_termination_msg=lambda msg: "TERMINATE" in msg["content"],
)

user_proxy.initiate_chat(
    assistant, message="What is the height of the Eiffel Tower? Only respond with the answer and terminate"
)
autogen.runtime_logging.stop()

This should create a runtime.log file in your current directory.