Using RetrieveChat for Retrieve Augmented Code Generation and Question Answering
Explore the use of AutoGen’s RetrieveChat for tasks like code generation from docstrings, answering complex questions with human feedback, and exploiting features like Update Context, custom prompts, and few-shot learning.
AutoGen offers conversable agents powered by LLM, tool or human, which can be used to perform tasks collectively via automated chat. This framework allows tool use and human participation through multi-agent conversation. Please find documentation about this feature here.
RetrieveChat is a conversational system for retrieval-augmented code
generation and question answering. In this notebook, we demonstrate how
to utilize RetrieveChat to generate code and answer questions based on
customized documentations that are not present in the LLM’s training
dataset. RetrieveChat uses the AssistantAgent
and
RetrieveUserProxyAgent
, which is similar to the usage of
AssistantAgent
and UserProxyAgent
in other notebooks (e.g.,
Automated Task Solving with Code Generation, Execution &
Debugging).
Essentially, RetrieveUserProxyAgent
implement a different auto-reply
mechanism corresponding to the RetrieveChat prompts.
Table of Contents
We’ll demonstrate six examples of using RetrieveChat for code generation and question answering:
- Example 1: Generate code based off docstrings w/o human feedback
- Example 2: Answer a question based off docstrings w/o human feedback
- Example 3: Generate code based off docstrings w/ human feedback
- Example 4: Answer a question based off docstrings w/ human feedback
- Example 5: Solve comprehensive QA problems with RetrieveChat’s
unique feature
Update Context
- Example 6: Solve comprehensive QA problems with customized prompt and few-shot learning
Some extra dependencies are needed for this notebook, which can be installed via pip:
For more information, please refer to the installation guide.
Set your API Endpoint
The
config_list_from_json
function loads a list of configurations from an environment variable or
a json file.
Learn more about configuring LLMs for agents here.
Construct agents for RetrieveChat
We start by initializing the AssistantAgent
and
RetrieveUserProxyAgent
. The system message needs to be set to “You are
a helpful assistant.” for AssistantAgent. The detailed instructions are
given in the user message. Later we will use the
RetrieveUserProxyAgent.message_generator
to combine the instructions
and a retrieval augmented generation task for an initial prompt to be
sent to the LLM assistant.
Example 1
Use RetrieveChat to help generate sample code and automatically run the code and fix errors if there is any.
Problem: Which API should I use if I want to use FLAML for a classification task and I want to train the model in 30 seconds. Use spark to parallel the training. Force cancel jobs if time limit is reached.
Example 2
Use RetrieveChat to answer a question that is not related to code generation.
Problem: Who is the author of FLAML?
Example 3
Use RetrieveChat to help generate sample code and ask for human-in-loop feedbacks.
Problem: how to build a time series forecasting model for stock price using FLAML?
Example 4
Use RetrieveChat to answer a question and ask for human-in-loop feedbacks.
Problem: Is there a function named tune_automl
in FLAML?
Example 5
Use RetrieveChat to answer questions for NaturalQuestion dataset.
First, we will create a new document collection which includes all the
contextual corpus. Then, we will choose some questions and utilize
RetrieveChat to answer them. For this particular example, we will be
using the gpt-3.5-turbo
model, and we will demonstrate RetrieveChat’s
feature of automatically updating context in case the documents
retrieved do not contain sufficient information.
In this example, questions were directly selected from the dataset.
RetrieveChat was able to answer the questions correctly in the first
attempt as the retrieved context contained the necessary information in
the first two cases. However, in the last three cases, the context with
the highest similarity to the question embedding did not contain the
required information to answer the question. As a result, the LLM model
responded with UPDATE CONTEXT
. With the unique and innovative ability
to update context in RetrieveChat, the agent automatically updated the
context and sent it to the LLM model again. After several rounds of this
process, the agent was able to generate the correct answer to the
questions.
Example 6
Use RetrieveChat to answer multi-hop questions for 2WikiMultihopQA dataset with customized prompt and few-shot learning.
First, we will create a new document collection which includes all the
contextual corpus. Then, we will choose some questions and utilize
RetrieveChat to answer them. For this particular example, we will be
using the gpt-3.5-turbo
model, and we will demonstrate RetrieveChat’s
feature of automatically updating context in case the documents
retrieved do not contain sufficient information. Moreover, we’ll
demonstrate how to use customized prompt and few-shot learning to
address tasks that are not pre-defined in RetrieveChat.