Use Cases
- Use cases
- Reference Agents
- Notebooks
- All Notebooks
- Solving Multiple Tasks in a Sequence of Chats
- Solving Multiple Tasks in a Sequence of Async Chats
- Using RetrieveChat for Retrieve Augmented Code Generation and Question Answering
- Interactive LLM Agent Dealing with Data Stream
- Agent Chat with Async Human Inputs
- Using RetrieveChat Powered by MongoDB Atlas for Retrieve Augmented Code Generation and Question Answering
- Auto Generated Agent Chat: Task Solving with Langchain Provided Tools as Functions
- Task Solving with Code Generation, Execution and Debugging
- Runtime Logging with AutoGen
- Agent Chat with Multimodal Models: DALLE and GPT-4V
- Using RetrieveChat with Qdrant for Retrieve Augmented Code Generation and Question Answering
- Enhanced Swarm Orchestration with AG2
- Translating Video audio using Whisper and GPT-3.5-turbo
- Agent Tracking with AgentOps
- Group Chat with Coder and Visualization Critic
- ReasoningAgent - Advanced LLM Reasoning with Multiple Search Strategies
- DeepResearchAgent
- Tools with Dependency Injection
- A Uniform interface to call different LLMs
- SQL Agent for Spider text-to-SQL benchmark
- DeepSeek: Adding Browsing Capabilities to AG2
- CaptainAgent
- Adding Browsing Capabilities to AG2
- Auto Generated Agent Chat: Teaching AI New Skills via Natural Language Interaction
- Writing a software application using function calls
- Auto Generated Agent Chat: Function Inception
- Agentic RAG workflow on tabular data from a PDF file
- RealtimeAgent in a Swarm Orchestration
- Using FalkorGraphRagCapability with agents for GraphRAG Question & Answering
- Making OpenAI Assistants Teachable
- Cross-Framework LLM Tool Integration with AG2
- RealtimeAgent with local websocket connection
- Using RetrieveChat Powered by PGVector for Retrieve Augmented Code Generation and Question Answering
- OpenAI Assistants in AutoGen
- Solving Complex Tasks with Nested Chats
- Auto Generated Agent Chat: GPTAssistant with Code Interpreter
- Group Chat with Customized Speaker Selection Method
- Structured output
- Agent Chat with custom model loading
- AutoBuild
- FSM - User can input speaker transition constraints
- Using Neo4j's graph database with AG2 agents for Question & Answering
- Use AutoGen in Databricks with DBRX
- Language Agent Tree Search
- Using a local Telemetry server to monitor a GraphRAG agent
- SocietyOfMindAgent
- Perform Research with Multi-Agent Group Chat
- Group Chat
- StateFlow: Build Workflows through State-Oriented Actions
- Agent Chat with Multimodal Models: LLaVA
- Trip planning with a FalkorDB GraphRAG agent using a Swarm
- Auto Generated Agent Chat: Collaborative Task Solving with Multiple Agents and Human Users
- Agent with memory using Mem0
- Preprocessing Chat History with `TransformMessages`
- AgentOptimizer: An Agentic Way to Train Your LLM Agent
- Using RetrieveChat Powered by Couchbase Capella for Retrieve Augmented Code Generation and Question Answering
- Agent Observability with OpenLIT
- Chatting with a teachable agent
- RealtimeAgent in a Swarm Orchestration
- Conversational Chess using non-OpenAI clients
- Solving Complex Tasks with A Sequence of Nested Chats
- Config loader utility functions
- Currency Calculator: Task Solving with Provided Tools as Functions
- Chat with OpenAI Assistant using function call in AutoGen: OSS Insights for Advanced GitHub Data Analysis
- Web Scraping using Apify Tools
- Agent with memory using Mem0
- Auto Generated Agent Chat: Collaborative Task Solving with Coding and Planning Agent
- Demonstrating the `AgentEval` framework using the task of solving math problems as an example
- RealtimeAgent with WebRTC connection
- Automatically Build Multi-agent System from Agent Library
- Groupchat with Llamaindex agents
- Using Guidance with AutoGen
- RealtimeAgent in a Swarm Orchestration using WebRTC
- Cross-Framework LLM Tool for CaptainAgent
- RealtimeAgent in a Swarm Orchestration
- Websockets: Streaming input and output using websockets
- Auto Generated Agent Chat: Solving Tasks Requiring Web Info
- Auto Generated Agent Chat: Task Solving with Code Generation, Execution, Debugging & Human Feedback
- Assistants with Azure Cognitive Search and Azure Identity
- (Legacy) Implement Swarm-style orchestration with GroupChat
- Auto Generated Agent Chat: Task Solving with Provided Tools as Functions
- Agentchat MathChat
- Small, Local Model (IBM Granite) Multi-Agent RAG
- Group Chat with Retrieval Augmented Generation
- Run a standalone AssistantAgent
- Nested Chats for Tool Use in Conversational Chess
- Swarm Orchestration with AG2
- OptiGuide with Nested Chats in AutoGen
- Usage tracking with AutoGen
- From Dad Jokes To Sad Jokes: Function Calling with GPTAssistantAgent
- Discord, Slack, and Telegram messaging tools
- RAG OpenAI Assistants in AutoGen
- Auto Generated Agent Chat: Group Chat with GPTAssistantAgent
- Task Solving with Provided Tools as Functions (Asynchronous Function Calls)
- Generate Dalle Images With Conversable Agents
- Chat Context Dependency Injection
- Solving Multiple Tasks in a Sequence of Chats with Different Conversable Agent Pairs
- Engaging with Multimodal Models: GPT-4V in AutoGen
- Mitigating Prompt hacking with JSON Mode in Autogen
- Using Neo4j's native GraphRAG SDK with AG2 agents for Question & Answering
- Supercharging Web Crawling with Crawl4AI
- WebSurferAgent
- RealtimeAgent with gemini client
- Community Gallery
Agent Observability with OpenLIT
Use OpenLIT to easily monitor AI agents in production with OpenTelemetry.
OpenLIT an open source product that helps developers build and manage AI agents in production, effectively helping them improve accuracy. As a self-hosted solution, it enables developers to experiment with LLMs, manage and version prompts, securely manage API keys, and provide safeguards against prompt injection and jailbreak attempts. It also includes built-in OpenTelemetry-native observability and evaluation for the complete GenAI stack (LLMs, Agents, vector databases, and GPUs).
For more info, check out the OpenLIT Repo
Adding OpenLIT to an existing AutoGen (Now AG2) service
To get started, you’ll need to install the OpenLIT library
OpenLIT uses OpenTelemetry to automatically instrument the AI Agent app when it’s initialized meaning your agent observability data like execution traces and metrics will be tracked in just one line of code.
pip install ag2 openlit
Collecting ag2
Downloading ag2-0.4-py3-none-any.whl.metadata (24 kB)
Collecting openlit
Downloading openlit-1.32.4-py3-none-any.whl.metadata (22 kB)
Collecting diskcache (from ag2)
Downloading diskcache-5.6.3-py3-none-any.whl.metadata (20 kB)
Collecting docker (from ag2)
Downloading docker-7.1.0-py3-none-any.whl.metadata (3.9 kB)
Collecting flaml (from ag2)
Downloading FLAML-2.3.2-py3-none-any.whl.metadata (16 kB)
Requirement already satisfied: openai>=1.3 in /usr/local/lib/python3.10/dist-packages (from ag2) (1.54.4)
Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from ag2) (24.2)
Requirement already satisfied: pydantic!=2.6.0,<3,>=1.10 in /usr/local/lib/python3.10/dist-packages (from ag2) (2.9.2)
Collecting python-dotenv (from ag2)
Downloading python_dotenv-1.0.1-py3-none-any.whl.metadata (23 kB)
Requirement already satisfied: termcolor in /usr/local/lib/python3.10/dist-packages (from ag2) (2.5.0)
Collecting tiktoken (from ag2)
Downloading tiktoken-0.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Requirement already satisfied: numpy<2.0.0,>=1.24.0 in /usr/local/lib/python3.10/dist-packages (from ag2) (1.26.4)
Collecting anthropic<0.22.0,>=0.21.0 (from openlit)
Downloading anthropic-0.21.3-py3-none-any.whl.metadata (17 kB)
Collecting boto3<2.0.0,>=1.34.0 (from openlit)
Downloading boto3-1.35.69-py3-none-any.whl.metadata (6.7 kB)
Collecting botocore<2.0.0,>=1.34.0 (from openlit)
Downloading botocore-1.35.69-py3-none-any.whl.metadata (5.7 kB)
Requirement already satisfied: opentelemetry-api<2.0.0,>=1.27.0 in /usr/local/lib/python3.10/dist-packages (from openlit) (1.28.2)
Collecting opentelemetry-exporter-otlp<2.0.0,>=1.27.0 (from openlit)
Downloading opentelemetry_exporter_otlp-1.28.2-py3-none-any.whl.metadata (2.3 kB)
Collecting opentelemetry-instrumentation<0.49,>=0.48b0 (from openlit)
Downloading opentelemetry_instrumentation-0.48b0-py3-none-any.whl.metadata (6.1 kB)
Requirement already satisfied: opentelemetry-sdk<2.0.0,>=1.27.0 in /usr/local/lib/python3.10/dist-packages (from openlit) (1.28.2)
Requirement already satisfied: requests<3.0.0,>=2.26.0 in /usr/local/lib/python3.10/dist-packages (from openlit) (2.32.3)
Collecting schedule<2.0.0,>=1.2.2 (from openlit)
Downloading schedule-1.2.2-py3-none-any.whl.metadata (3.9 kB)
Collecting tiktoken (from ag2)
Downloading tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB)
Collecting xmltodict<0.14.0,>=0.13.0 (from openlit)
Downloading xmltodict-0.13.0-py2.py3-none-any.whl.metadata (7.7 kB)
Requirement already satisfied: anyio<5,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.22.0,>=0.21.0->openlit) (3.7.1)
Requirement already satisfied: distro<2,>=1.7.0 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.22.0,>=0.21.0->openlit) (1.9.0)
Requirement already satisfied: httpx<1,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.22.0,>=0.21.0->openlit) (0.27.2)
Requirement already satisfied: sniffio in /usr/local/lib/python3.10/dist-packages (from anthropic<0.22.0,>=0.21.0->openlit) (1.3.1)
Requirement already satisfied: tokenizers>=0.13.0 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.22.0,>=0.21.0->openlit) (0.20.3)
Requirement already satisfied: typing-extensions<5,>=4.7 in /usr/local/lib/python3.10/dist-packages (from anthropic<0.22.0,>=0.21.0->openlit) (4.12.2)
Collecting jmespath<2.0.0,>=0.7.1 (from boto3<2.0.0,>=1.34.0->openlit)
Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting s3transfer<0.11.0,>=0.10.0 (from boto3<2.0.0,>=1.34.0->openlit)
Downloading s3transfer-0.10.4-py3-none-any.whl.metadata (1.7 kB)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.10/dist-packages (from botocore<2.0.0,>=1.34.0->openlit) (2.8.2)
Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /usr/local/lib/python3.10/dist-packages (from botocore<2.0.0,>=1.34.0->openlit) (2.2.3)
Requirement already satisfied: jiter<1,>=0.4.0 in /usr/local/lib/python3.10/dist-packages (from openai>=1.3->ag2) (0.7.1)
Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from openai>=1.3->ag2) (4.66.6)
Requirement already satisfied: deprecated>=1.2.6 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-api<2.0.0,>=1.27.0->openlit) (1.2.15)
Requirement already satisfied: importlib-metadata<=8.5.0,>=6.0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-api<2.0.0,>=1.27.0->openlit) (8.5.0)
Collecting opentelemetry-exporter-otlp-proto-grpc==1.28.2 (from opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit)
Downloading opentelemetry_exporter_otlp_proto_grpc-1.28.2-py3-none-any.whl.metadata (2.2 kB)
Collecting opentelemetry-exporter-otlp-proto-http==1.28.2 (from opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit)
Downloading opentelemetry_exporter_otlp_proto_http-1.28.2-py3-none-any.whl.metadata (2.2 kB)
Requirement already satisfied: googleapis-common-protos~=1.52 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-exporter-otlp-proto-grpc==1.28.2->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit) (1.66.0)
Requirement already satisfied: grpcio<2.0.0,>=1.63.2 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-exporter-otlp-proto-grpc==1.28.2->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit) (1.68.0)
Collecting opentelemetry-exporter-otlp-proto-common==1.28.2 (from opentelemetry-exporter-otlp-proto-grpc==1.28.2->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit)
Downloading opentelemetry_exporter_otlp_proto_common-1.28.2-py3-none-any.whl.metadata (1.8 kB)
Collecting opentelemetry-proto==1.28.2 (from opentelemetry-exporter-otlp-proto-grpc==1.28.2->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit)
Downloading opentelemetry_proto-1.28.2-py3-none-any.whl.metadata (2.3 kB)
Collecting protobuf<6.0,>=5.0 (from opentelemetry-proto==1.28.2->opentelemetry-exporter-otlp-proto-grpc==1.28.2->opentelemetry-exporter-otlp<2.0.0,>=1.27.0->openlit)
Downloading protobuf-5.28.3-cp38-abi3-manylinux2014_x86_64.whl.metadata (592 bytes)
Requirement already satisfied: setuptools>=16.0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-instrumentation<0.49,>=0.48b0->openlit) (75.1.0)
Requirement already satisfied: wrapt<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-instrumentation<0.49,>=0.48b0->openlit) (1.16.0)
Requirement already satisfied: opentelemetry-semantic-conventions==0.49b2 in /usr/local/lib/python3.10/dist-packages (from opentelemetry-sdk<2.0.0,>=1.27.0->openlit) (0.49b2)
Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=2.6.0,<3,>=1.10->ag2) (0.7.0)
Requirement already satisfied: pydantic-core==2.23.4 in /usr/local/lib/python3.10/dist-packages (from pydantic!=2.6.0,<3,>=1.10->ag2) (2.23.4)
Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.26.0->openlit) (3.4.0)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.26.0->openlit) (3.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.26.0->openlit) (2024.8.30)
Requirement already satisfied: regex>=2022.1.18 in /usr/local/lib/python3.10/dist-packages (from tiktoken->ag2) (2024.9.11)
Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<5,>=3.5.0->anthropic<0.22.0,>=0.21.0->openlit) (1.2.2)
Requirement already satisfied: httpcore==1.* in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->anthropic<0.22.0,>=0.21.0->openlit) (1.0.7)
Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/lib/python3.10/dist-packages (from httpcore==1.*->httpx<1,>=0.23.0->anthropic<0.22.0,>=0.21.0->openlit) (0.14.0)
Requirement already satisfied: zipp>=3.20 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata<=8.5.0,>=6.0->opentelemetry-api<2.0.0,>=1.27.0->openlit) (3.21.0)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<2.0.0,>=1.34.0->openlit) (1.16.0)
Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /usr/local/lib/python3.10/dist-packages (from tokenizers>=0.13.0->anthropic<0.22.0,>=0.21.0->openlit) (0.26.2)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic<0.22.0,>=0.21.0->openlit) (3.16.1)
Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic<0.22.0,>=0.21.0->openlit) (2024.10.0)
Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic<0.22.0,>=0.21.0->openlit) (6.0.2)
Downloading ag2-0.4-py3-none-any.whl (366 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 366.1/366.1 kB 8.1 MB/s eta 0:00:00
Downloading openlit-1.32.4-py3-none-any.whl (229 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.1/229.1 kB 15.1 MB/s eta 0:00:00
Downloading anthropic-0.21.3-py3-none-any.whl (851 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 851.6/851.6 kB 8.5 MB/s eta 0:00:00
Downloading boto3-1.35.69-py3-none-any.whl (139 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 139.2/139.2 kB 9.5 MB/s eta 0:00:00
Downloading botocore-1.35.69-py3-none-any.whl (13.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.0/13.0 MB 57.1 MB/s eta 0:00:00
Downloading opentelemetry_exporter_otlp-1.28.2-py3-none-any.whl (7.0 kB)
Downloading opentelemetry_exporter_otlp_proto_grpc-1.28.2-py3-none-any.whl (18 kB)
Downloading opentelemetry_exporter_otlp_proto_http-1.28.2-py3-none-any.whl (17 kB)
Downloading opentelemetry_exporter_otlp_proto_common-1.28.2-py3-none-any.whl (18 kB)
Downloading opentelemetry_proto-1.28.2-py3-none-any.whl (55 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.8/55.8 kB 4.3 MB/s eta 0:00:00
Downloading opentelemetry_instrumentation-0.48b0-py3-none-any.whl (29 kB)
Downloading schedule-1.2.2-py3-none-any.whl (12 kB)
Downloading tiktoken-0.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 42.3 MB/s eta 0:00:00
Downloading xmltodict-0.13.0-py2.py3-none-any.whl (10.0 kB)
Downloading diskcache-5.6.3-py3-none-any.whl (45 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.5/45.5 kB 3.0 MB/s eta 0:00:00
Downloading docker-7.1.0-py3-none-any.whl (147 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 147.8/147.8 kB 8.6 MB/s eta 0:00:00
Downloading FLAML-2.3.2-py3-none-any.whl (313 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 313.9/313.9 kB 19.8 MB/s eta 0:00:00
Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB)
Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.2/83.2 kB 6.9 MB/s eta 0:00:00
Downloading protobuf-5.28.3-cp38-abi3-manylinux2014_x86_64.whl (316 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 316.6/316.6 kB 21.0 MB/s eta 0:00:00
Installing collected packages: xmltodict, schedule, python-dotenv, protobuf, jmespath, flaml, diskcache, tiktoken, opentelemetry-proto, docker, botocore, s3transfer, opentelemetry-instrumentation, opentelemetry-exporter-otlp-proto-common, boto3, anthropic, ag2, opentelemetry-exporter-otlp-proto-http, opentelemetry-exporter-otlp-proto-grpc, opentelemetry-exporter-otlp, openlit
Attempting uninstall: protobuf
Found existing installation: protobuf 4.25.5
Uninstalling protobuf-4.25.5:
Successfully uninstalled protobuf-4.25.5
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorflow 2.17.1 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3, but you have protobuf 5.28.3 which is incompatible.
tensorflow-metadata 1.13.1 requires protobuf<5,>=3.20.3, but you have protobuf 5.28.3 which is incompatible.
Successfully installed ag2-0.4 anthropic-0.21.3 boto3-1.35.69 botocore-1.35.69 diskcache-5.6.3 docker-7.1.0 flaml-2.3.2 jmespath-1.0.1 openlit-1.32.4 opentelemetry-exporter-otlp-1.28.2 opentelemetry-exporter-otlp-proto-common-1.28.2 opentelemetry-exporter-otlp-proto-grpc-1.28.2 opentelemetry-exporter-otlp-proto-http-1.28.2 opentelemetry-instrumentation-0.48b0 opentelemetry-proto-1.28.2 protobuf-5.28.3 python-dotenv-1.0.1 s3transfer-0.10.4 schedule-1.2.2 tiktoken-0.7.0 xmltodict-0.13.0
import openlit
from autogen import AssistantAgent, UserProxyAgent
openlit.init()
OpenLIT will now start automatically tracking
- LLM prompts and completions
- Token usage and costs
- Agent names and actions
- Tool usage
- Errors
Lets look at a simple chat example
import openlit
openlit.init()
import os
llm_config = {"model": "gpt-4", "api_key": os.environ["OPENAI_API_KEY"]}
assistant = AssistantAgent("assistant", llm_config=llm_config)
user_proxy = UserProxyAgent("user_proxy", code_execution_config=False)
# Start the chat
user_proxy.initiate_chat(
assistant,
message="Tell me a joke about NVDA and TESLA stock prices.",
)
user_proxy (to assistant):
Tell me a joke about NVDA and TESLA stock prices.
--------------------------------------------------------------------------------
assistant (to user_proxy):
Why don't NVDA and TESLA stock prices ever get coffee together?
Because whenever they heat up, they always take a steep drop before they can cool down!
I hope this brings a smile to your face. Investing in stocks can be a rollercoaster sometimes. Please note that this is humor and doesn't reflect the actual dynamics of these companies' stock prices. TERMINATE.
--------------------------------------------------------------------------------
Replying as user_proxy. Provide feedback to assistant. Press enter to skip and use auto-reply, or type 'exit' to end the conversation: hi
user_proxy (to assistant):
hi
--------------------------------------------------------------------------------
{
"name": "openai.chat.completions",
"context": {
"trace_id": "0x35ae5952626492e432ae25af5bf92daa",
"span_id": "0x44103383aa51d1b1",
"trace_state": "[]"
},
"kind": "SpanKind.CLIENT",
"parent_id": null,
"start_time": "2024-11-21T01:53:47.597241Z",
"end_time": "2024-11-21T01:53:48.506758Z",
"status": {
"status_code": "OK"
},
"attributes": {
"telemetry.sdk.name": "openlit",
"gen_ai.system": "openai",
"gen_ai.operation.name": "chat",
"gen_ai.endpoint": "openai.chat.completions",
"gen_ai.response.id": "chatcmpl-AVqg7QSYB8CpEN1I5PYT1laBcds9S",
"gen_ai.environment": "default",
"gen_ai.application_name": "default",
"gen_ai.request.model": "gpt-4",
"gen_ai.request.top_p": 1.0,
"gen_ai.request.max_tokens": -1,
"gen_ai.request.user": "",
"gen_ai.request.temperature": 1.0,
"gen_ai.request.presence_penalty": 0.0,
"gen_ai.request.frequency_penalty": 0.0,
"gen_ai.request.seed": "",
"gen_ai.request.is_stream": false,
"gen_ai.usage.input_tokens": 580,
"gen_ai.usage.output_tokens": 9,
"gen_ai.usage.total_tokens": 589,
"gen_ai.response.finish_reasons": [
"stop"
],
"gen_ai.usage.cost": 0.017939999999999998
},
"events": [
{
"name": "gen_ai.content.prompt",
"timestamp": "2024-11-21T01:53:48.506257Z",
"attributes": {
"gen_ai.prompt": "system: You are a helpful AI assistant.\nSolve tasks using your coding and language skills.\nIn the following cases, suggest python code (in a python coding block) or shell script (in a sh coding block) for the user to execute.\n 1. When you need to collect info, use the code to output the info you need, for example, browse or search the web, download/read a file, print the content of a webpage or a file, get the current date/time, check the operating system. After sufficient info is printed and the task is ready to be solved based on your language skill, you can solve the task by yourself.\n 2. When you need to perform some task with code, use the code to perform the task and output the result. Finish the task smartly.\nSolve the task step by step if you need to. If a plan is not provided, explain your plan first. Be clear which step uses code, and which step uses your language skill.\nWhen using code, you must indicate the script type in the code block. The user cannot provide any other feedback or perform any other action beyond executing the code you suggest. The user can't modify your code. So do not suggest incomplete code which requires users to modify. Don't use a code block if it's not intended to be executed by the user.\nIf you want the user to save the code in a file before executing it, put # filename: <filename> inside the code block as the first line. Don't include multiple code blocks in one response. Do not ask users to copy and paste the result. Instead, use 'print' function for the output when relevant. Check the execution result returned by the user.\nIf the result indicates there is an error, fix the error and output the code again. Suggest the full code instead of partial code or code changes. If the error can't be fixed or if the task is not solved even after the code is executed successfully, analyze the problem, revisit your assumption, collect additional info you need, and think of a different approach to try.\nWhen you find an answer, verify the answer carefully. Include verifiable evidence in your response if possible.\nReply \"TERMINATE\" in the end when everything is done.\n \nuser: Tell me a joke about NVDA and TESLA stock prices.\nassistant: Why don't NVDA and TESLA stock prices ever get coffee together?\n\nBecause whenever they heat up, they always take a steep drop before they can cool down! \n\nI hope this brings a smile to your face. Investing in stocks can be a rollercoaster sometimes. Please note that this is humor and doesn't reflect the actual dynamics of these companies' stock prices. TERMINATE.\nuser: hi"
}
},
{
"name": "gen_ai.content.completion",
"timestamp": "2024-11-21T01:53:48.506314Z",
"attributes": {
"gen_ai.completion": "Hello! How can I assist you today?"
}
}
],
"links": [],
"resource": {
"attributes": {
"telemetry.sdk.language": "python",
"telemetry.sdk.name": "openlit",
"telemetry.sdk.version": "1.28.2",
"service.name": "default",
"deployment.environment": "default"
},
"schema_url": ""
}
}
assistant (to user_proxy):
Hello! How can I assist you today?
--------------------------------------------------------------------------------
Replying as user_proxy. Provide feedback to assistant. Press enter to skip and use auto-reply, or type 'exit' to end the conversation: exit
ChatResult(chat_id=None, chat_history=[{'content': 'Tell me a joke about NVDA and TESLA stock prices.', 'role': 'assistant', 'name': 'user_proxy'}, {'content': "Why don't NVDA and TESLA stock prices ever get coffee together?\n\nBecause whenever they heat up, they always take a steep drop before they can cool down! \n\nI hope this brings a smile to your face. Investing in stocks can be a rollercoaster sometimes. Please note that this is humor and doesn't reflect the actual dynamics of these companies' stock prices. TERMINATE.", 'role': 'user', 'name': 'assistant'}, {'content': 'hi', 'role': 'assistant', 'name': 'user_proxy'}, {'content': 'Hello! How can I assist you today?', 'role': 'user', 'name': 'assistant'}], summary='Hello! How can I assist you today?', cost={'usage_including_cached_inference': {'total_cost': 0.03731999999999999, 'gpt-4-0613': {'cost': 0.03731999999999999, 'prompt_tokens': 1068, 'completion_tokens': 88, 'total_tokens': 1156}}, 'usage_excluding_cached_inference': {'total_cost': 0.017939999999999998, 'gpt-4-0613': {'cost': 0.017939999999999998, 'prompt_tokens': 580, 'completion_tokens': 9, 'total_tokens': 589}}}, human_input=['hi', 'exit'])
Sending Traces and metrics to OpenLIT
By default, OpenLIT generates OpenTelemetry traces and metrics that are logged to your console. To set up a detailed monitoring environment, this guide outlines how to deploy OpenLIT and direct all traces and metrics there. You also have the flexibility to send the telemetry data to any OpenTelemetry-compatible endpoint, such as Grafana, Jaeger, or DataDog.
Deploy OpenLIT Stack
-
Clone the OpenLIT Repository
Open your terminal or command line and execute:
git clone git@github.com:openlit/openlit.git
-
Host it Yourself with Docker
Deploy and start OpenLIT using the command:
docker compose up -d
For instructions on installing in Kubernetes using Helm, refer to the Kubernetes Helm installation guide.
Configure the telemetry data destination as follows:
Purpose | Parameter/Environment Variable | For Sending to OpenLIT |
---|---|---|
Send data to an HTTP OTLP endpoint | otlp_endpoint or OTEL_EXPORTER_OTLP_ENDPOINT |
"http://127.0.0.1:4318" |
Authenticate telemetry backends | otlp_headers or OTEL_EXPORTER_OTLP_HEADERS |
Not required by default |
💡 Info: If the
otlp_endpoint
orOTEL_EXPORTER_OTLP_ENDPOINT
is not provided, the OpenLIT SDK will output traces directly to your console, which is recommended during the development phase.
Visualize and Optimize!
With the Observability data now being collected and sent to OpenLIT, the next step is to visualize and analyze this data to get insights into your AI application’s performance, behavior, and identify areas of improvement.
Just head over to OpenLIT at 127.0.0.1:3000
on your browser to start
exploring. You can login using the default credentials - Email:
user@openlit.io
- Password: openlituser