Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cannot generate TestDataset more than once #848

Closed
njcolvin opened this issue Apr 5, 2024 · 6 comments · May be fixed by #1001
Closed

Cannot generate TestDataset more than once #848

njcolvin opened this issue Apr 5, 2024 · 6 comments · May be fixed by #1001
Labels
bug Something isn't working

Comments

@njcolvin
Copy link

njcolvin commented Apr 5, 2024 •
edited
Loading

[x] I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
Calling TestsetGenerator.generate_with_langchain_docs() more than once results in documents from the first call being used in subsequent calls.

Ragas version: 0.1.5
Python version: 3.11

Code to Reproduce

from langchain.document_loaders.directory import DirectoryLoader

loader = DirectoryLoader("your-directory") # contains some .txt file
documents = loader.load()

from ragas.testset.generator import TestsetGenerator
from ragas.testset.evolutions import simple, reasoning, multi_context
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# generator with openai models
generator_llm = ChatOpenAI(model="gpt-3.5-turbo")
critic_llm = ChatOpenAI(model="gpt-4")
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    embeddings
)

# generate testset
testset = generator.generate_with_langchain_docs(documents, test_size=1, distributions={simple: 1})

loader2 = DirectoryLoader("your-directory-2") # contains some other .txt file
documents2 = loader2.load()
generator_llm2 = ChatOpenAI(model="gpt-3.5-turbo")
critic_llm2 = ChatOpenAI(model="gpt-4")
embeddings2 = OpenAIEmbeddings(model="text-embedding-3-small")

generator2 = TestsetGenerator.from_langchain(
    generator_llm2,
    critic_llm2,
    embeddings2
)
testset2 = generator2.generate_with_langchain_docs(documents2, test_size=1, distributions={simple: 1})

Expected behavior
I expect testset2 to contain a question about the .txt file in your-directory-2, however it contains a question about the .txt file in your-directory.

Additional context
None.
@njcolvin njcolvin added the bug Something isn't working label Apr 5, 2024
@mingjun1120
Copy link

mingjun1120 commented Apr 5, 2024 •
edited
Loading

I am experiencing the same issue. I was only able to generate 3 test questions.

image

@njcolvin
Copy link
Author

njcolvin commented Apr 5, 2024 •
edited
Loading

AFAIK, the test_size parameter refers to the number of candidate questions, so it acts as a maximum but not a guaranteed amount. My question refers to calling generate() or generate_with_langchain_docs() multiple times using different documents. If you do this, the questions will be about the first documents passed for all calls. This is a memory management issue most likely caused by the InMemoryDocumentStore class from what I can tell.

@jjmachan
Copy link
Member

jjmachan commented Apr 8, 2024

plus 1 on what @njcolvin said.

The "evolutionary process" (details here) ragas uses for testset generation reduces the number of actual contexts you can use to generate question. the filter might filter out most of the documents because you can't generation high quality questions from it.

it also depends on the LLMs you use too

how many documents are there in documents2 @njcolvin

I can also spend some time with you on a call to debug further if you like 🙂

@njcolvin
Copy link
Author

Hi @jjmachan I appreciate the response. In the above example I believe I used Biden's state of the union for documents and the first 1000 lines of tinyshakespeare for documents2. I am unsure exactly how many documents were created from these sources.

You might be right about the filter discarding documents2 as it seemed to do this and then bring back the old documents. I was able to make it work for my application by calling the generation code with different documents each time using the subprocess module.

Thank you for offering to debug this with me. I am interested, and will follow up as soon as I can.

@Manav916
Copy link

Im having the same issue as @njcolvin. Any update @jjmachan?

@Kevin-JiXu
Copy link

I have the same issue. I will appreciate it if you udate.

@dosubot dosubot bot added the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label May 20, 2024
@Manav916 Manav916 mentioned this issue May 31, 2024
Open
@dosubot dosubot bot closed this as not planned Won't fix, can't repro, duplicate, stale Jun 1, 2024
@dosubot dosubot bot removed the stale Issue has not had recent activity or appears to be solved. Stale issues will be automatically closed label Jun 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(testset): for generating testset with new docs Manav916/ragas
5 participants
@jjmachan @njcolvin @Kevin-JiXu @mingjun1120 @Manav916