Issue while trying to use RAGAS to evaluate RAG application

[adhsay]

Hello,

If you can kindly help with the issue I am seeing.
I am trying to evaluate my RAG application using giskard and RAGAS,but on trying to do so I am getting the following error:

Giskard_Testing\giskard\rag\metrics\correctness.py", line 77, in call
out = llm_client.complete(
^^^^^^^^^^^^^^^^^^^
AttributeError: 'AzureChatOpenAI' object has no attribute 'complete'

I am using as seen above, AzureChatOpenAI as LLM.I see the same error even if I use ChatOpenAI.

Can you kindly please help on probable solution to this or where am I going wrong.
Thanks in advance.

[pierlj]

Hi, did you follow the instructions from this page to setup your LLM client?

Also, to better understand your issue, can you provide the code you ran to get this error?

[adhsay]

Hello @pierlj , thank you for your reply.

This is the code snippet I am trying to use:

os.environ['AZURE_OPENAI_API_KEY'] = 'key'
os.environ['AZURE_OPENAI_ENDPOINT'] = 'endpoint'
os.environ['OPENAI_API_VERSION'] = 'version'

#openAI
#giskard.llm.set_llm_api("openai")
#oc = OpenAIClient(model="gpt-4-turbo-preview")
#giskard.llm.set_default_client(oc)

You'll need to provide the name of the model that you've deployed

Beware, the model provided must be capable of using function calls

set_llm_model('gpt-35-turbo-16k')

from giskard.rag import generate_testset, KnowledgeBase

import chardet

r_file = open("state_of_the_union.txt", 'rb').read()
result = chardet.detect(r_file)
charenc = result['encoding']
print(charenc)

df = pd.read_csv("state_of_the_union.txt", sep="\t",encoding='Windows-1252')
knowledge_base = KnowledgeBase.from_pandas(df)

Generate a testset with 10 questions & answers for each question types (this will take a while)

testset = generate_testset(
knowledge_base,
num_questions=2,
agent_description="A customer support chatbot for company X", # helps generating better questions
)

Save the generated testset

testset.save("my_testset.json")

Convert it to a pandas dataframe

df = testset.to_pandas()
#df.to_csv("my_testset.csv")
#print(df)
#print(df["question"])

##########################################################################################################

pages = []
loader = TextLoader("./state_of_the_union.txt")

loader.load_and_split()

local_pages = loader.load_and_split() # load and split the pdf into pages
print(len(local_pages))
pages.extend(local_pages)

split the pages into chunks

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=50)
splits = text_splitter.split_documents(pages)

Create a vector store for the sample data

persist_directory = 'data/chroma/'
#!rm -rf ./data/chroma # remove old database files if any

#embedding = OpenAIEmbeddings()
embedding = AzureOpenAIEmbeddings(
openai_api_version="version",
api_key="key",
azure_endpoint="endpoint",
azure_deployment="text-embedding-ada-002",
)

vectordb = Chroma.from_documents(
documents=splits,
embedding=embedding,
persist_directory=persist_directory
)
vectordb.persist()

retriever = vectordb.as_retriever()

Define LLM

#llm = ChatOpenAI(model_name="gpt-3.5-turbo", temperature=0)

llm = AzureChatOpenAI(
openai_api_version="version",
api_key="key",
azure_endpoint="endpoint",
azure_deployment="gpt-4",
)

Define prompt template

prompt_template = """You are an assistant for question-answering tasks.
Use the following pieces of retrieved context to answer the question.
If you don't know the answer, just say that you don't know.
Question: {question}
Context: {context}
Answer:
"""

prompt = ChatPromptTemplate.from_template(prompt_template)

retrieval = RunnableParallel(
{"context": retriever, "question": RunnablePassthrough()}
)

chain = retrieval | prompt | llm | StrOutputParser()

from giskard.rag import evaluate
answers = []
context = []
queries = df["question"].tolist()
for query in queries:
#question = str(df["question"])
answers.append(chain.invoke(query))
context.append([docs.page_content for docs in retriever.get_relevant_documents(query)])
#print(f"answer is: {answers}")

Run the evaluation and get a report

report = evaluate(answers, testset=testset, knowledge_base=knowledge_base)
report.correctness_by_topic()
report.correctness_by_question_type()
report.failures
report.to_html("rag_eval_report.html")

from giskard.rag.metrics.ragas_metrics import ragas_context_recall, ragas_faithfulness, ragas_context_precision, ragas_answer_relevancy
#ground_truth = df['ground_truth'].tolist()

from giskard.rag.base import AgentAnswer

aa= AgentAnswer(message=answers,documents=context)

report1 = evaluate(
answers,
testset=testset,
knowledge_base=knowledge_base,
#metrics=[ragas_answer_relevancy],
llm_client=llm,
)
report1.to_html("ragas_eval_report_giskard.html")

[pierlj]

You get this error because you pass an instance of AzureChatOpenAI to the evaluate function. This should instead be an instance of giskard LLMClient. You set the default model to gpt-35-turbo-16k above, so I guess you want to use gpt 4 only for the evaluation. To do so, you can create a new LLMClient as follows:

from giskard.llm.client.openai import OpenAIClient
from openai import AzureOpenAI

llm_client = OpenAIClient("gpt-4", AzureOpenAI())

Since you configured the azure env variable properly, this should work. Another possibility is to change the default model used by giskard library with set_default_llm and don't pass anything to the evaluate function, but it changes the model globally, so if you want to use it anywhere else, it might not be convenient.

[adhsay]

Thank you @pierlj for your kind help on this, it works now.

[adhsay]

One more query please @pierlj , can you kindly give an example of how I can use "retrieved_documents" to evaluate RAGAS instead of using functions like get_answer_fn.Thanks in advance.

[pierlj]

To compute the ragas metrics, you need to define an answer_fn that returns an instance of AgentAnswer. The retrieved_document argument from the evaluate function is deprecated, but you can pass a list of AgentAnswer in place of a list of strings as the answers. I will update the docs as it is still mentioned there.

Here is an example you can follow:

from giskard.rag import AgentAnwer

def get_answer_fn(question: str, history=None) -> str:
    """A function representing your RAG agent."""
    # Format appropriately the history for your RAG agent
    messages = history if history else []
    messages.append({"role": "user", "content": question})

    # Get the answer and the documents
    agent_output = get_answer_from_agent(messages)

    # Following llama_index syntax, you can get the answer and the retrieved documents
    answer = agent_output.text
    documents = agent_output.source_nodes

    # Instead of returning a simple string, we return the AgentAnswer object which
    # allows us to specify the retrieved context which is used by RAGAS metrics
    return AgentAnswer(
        message=answer,
        documents=documents
    )