Tried Generation Test Set from Together APIs and Hugging Face Embeddings

[Eknathabhiram]

[ ] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
Tried Generation Test Set from Together APIs and Hugging Face Embeddings

Ragas version:0.1.7
Python version: 3.9

Code to Reproduce

import os
from langchain_community.document_loaders import PyPDFDirectoryLoader
from langchain.embeddings import SentenceTransformerEmbeddings


from ragas.testset.generator import TestsetGenerator
from langchain_together import Together
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

generator_llm = Together(
        model="mistralai/Mistral-7B-Instruct-v0.2",
        together_api_key=os.getenv("TOGETHER_API_KEY")
    )
critic_llm = Together(
        model="meta-llama/Llama-3-70b-chat-hf",
        together_api_key=os.getenv("TOGETHER_API_KEY")
    )


embeddings = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

loader = PyPDFDirectoryLoader("/home/[email protected]/Documents/RAG Pipeline/new_demo/temp_pdfs")
documents = loader.load()
# generate testset

testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
print(testset)
testset.to_pandas().to_csv()

Error trace

Expected behavior

Filename and doc_id are the same for all nodes.                                                                     
Generating:   0%|                                                                            | 0/10 [00:00<?, ?it/s]Failed to parse output. Returning None.
Generating:   0%|                                                                            | 0/10 [00:10<?, ?it/s]
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    self.run()
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 96, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 84, in _aresults
    raise e
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 79, in _aresults
    r = await future
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py", line 611, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 38, in sema_coro
    return await coro
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 142, in evolve
    ) = await self._aevolve(current_tries, current_nodes)
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 290, in _aevolve
    passed = await self.node_filter.filter(merged_node)
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/filters.py", line 60, in filter
    output["score"] = sum(output.values()) / len(output.values())
ZeroDivisionError: division by zero
Traceback (most recent call last):
  File "/home/[email protected]/Documents/RAG Pipeline/new_demo/test.py", line 34, in <module>
    testset = generator.generate_with_langchain_docs(documents, test_size=10, distributions={simple: 0.5, reasoning: 0.25, multi_context: 0.25})
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/generator.py", line 179, in generate_with_langchain_docs
    return self.generate(
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/generator.py", line 274, in generate
    raise ExceptionInRunner()
ragas.exceptions.ExceptionInRunner: The runner thread which was running the jobs raised an exeception. Read the traceback above to debug it. You can also pass `raise_exceptions=False` incase you want to show only a warning message instead.
Exception ignored in: <coroutine object Executor.wrap_callable_with_index.<locals>.wrapped_callable_async at 0x7f2d728700c0>
Traceback (most recent call last):
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 142, in evolve
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 545, in _aevolve
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 373, in _acomplex_evolution
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 290, in _aevolve
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/filters.py", line 58, in filter
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/llms/output_parser.py", line 67, in aparse
KeyError: 'idle_for'
Exception ignored in: <coroutine object Executor.wrap_callable_with_index.<locals>.wrapped_callable_async at 0x7f2d728709c0>
Traceback (most recent call last):
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 142, in evolve
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 460, in _aevolve
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 290, in _aevolve
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/filters.py", line 58, in filter
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/llms/output_parser.py", line 67, in aparse
KeyError: 'idle_for'
Exception ignored in: <coroutine object Executor.wrap_callable_with_index.<locals>.wrapped_callable_async at 0x7f2d72870bc0>
Traceback (most recent call last):
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 142, in evolve
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 545, in _aevolve
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 373, in _acomplex_evolution
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 306, in _aevolve
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/filters.py", line 89, in filter
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/llms/base.py", line 92, in generate
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 142, in async_wrapped
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 58, in __call__
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 110, in iter
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 78, in inner
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/__init__.py", line 420, in next_action
KeyError: 'idle_for'
Exception ignored in: <coroutine object Executor.wrap_callable_with_index.<locals>.wrapped_callable_async at 0x7f2d729a4cc0>
Traceback (most recent call last):
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 142, in evolve
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/testset/evolutions.py", line 298, in _aevolve
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/llms/base.py", line 92, in generate
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 142, in async_wrapped
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 58, in __call__
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 110, in iter
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/_asyncio.py", line 78, in inner
  File "/home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/tenacity/__init__.py", line 420, in next_action
KeyError: 'idle_for'
Task was destroyed but it is pending!
task: <Task pending name='Task-677' coro=<as_completed.<locals>.sema_coro() running at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d728b7160>()]> cb=[as_completed.<locals>._on_completion() at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-675' coro=<as_completed.<locals>.sema_coro() running at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d72910ca0>()]> cb=[as_completed.<locals>._on_completion() at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-673' coro=<as_completed.<locals>.sema_coro() running at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d728b70a0>()]> cb=[as_completed.<locals>._on_completion() at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-681' coro=<as_completed.<locals>.sema_coro() running at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d72970b50>()]> cb=[as_completed.<locals>._on_completion() at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-674' coro=<as_completed.<locals>.sema_coro() running at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d726395e0>()]> cb=[as_completed.<locals>._on_completion() at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-678' coro=<as_completed.<locals>.sema_coro() running at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d72903670>()]> cb=[as_completed.<locals>._on_completion() at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-676' coro=<as_completed.<locals>.sema_coro() running at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d726352b0>()]> cb=[as_completed.<locals>._on_completion() at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-679' coro=<as_completed.<locals>.sema_coro() running at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d727dad90>()]> cb=[as_completed.<locals>._on_completion() at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>
Task was destroyed but it is pending!
task: <Task pending name='Task-682' coro=<as_completed.<locals>.sema_coro() running at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/site-packages/ragas/executor.py:38> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f2d72957640>()]> cb=[as_completed.<locals>._on_completion() at /home/[email protected]/anaconda3/envs/rag-env/lib/python3.9/asyncio/tasks.py:598]>

Additional context
Add any other context about the problem here.

[ciekawy]

Have a look at some changes that allowed me to use together ai:

#936

[jjmachan]

hey @Eknathabhiram, this is a class of bugs that are due to how we interface with different LLM providers. Doing a broader fix for this soon 🙂

hey @ciekawy thank you so much for putting a fix for this and my sinciear apologies for not taking a closer look at this earlier 🙏🏽
will take a look at this today and leave a review and get it merged soon 🙂 ❤️