Test the evaluation harness on RAG parameter tuning

[mrm1001]

Context

The evaluation harness is going to be available for our users in the haystack-experimental package. In order for us to decide whether we want to keep this feature or not, and whether it needs any improvements, we would like to first test it internally before we collect feedback from our users.

Task description

Use the evaluation harness to optimise a RAG pipeline using parameter search.

Task outcome

• Discover and find any bugs or UX issues
• Create shareable code that can later be used to get feedback from users. This code will live in the evaluation repository.

[davidsbatista]

The Harness is not the best tool for parameter search and much more a beginner level tool - I used it to showcase Basic RAG vs HyDE approach, and in the process reported a few bugs and issues which were and are being taken care of.

[davidsbatista]

a working example with the Harness, showcasing a baseline RAG vs the HyDE technique is referenced here:

• https://github.com/deepset-ai/haystack-evaluation/tree/main/evaluations

and the full code here

• https://github.com/deepset-ai/haystack-evaluation/blob/main/evaluations/evaluation_aragog_harness.py