Auto-merging Retrieval

[davidsbatista]

Auto-merging retrieval aims to combine (or merge) information from multiple sources or segments of text to create a more comprehensive and contextually relevant response to a query.

This approach is particularly useful when no single document or segment fully answers the query but rather the answer lies in combining information from multiple sources.

It allows smaller chunks to be merged into bigger parent chunks. It does this via the following steps:

1. Define a hierarchy of smaller chunks linked to parent chunks.
2. If the set of smaller chunks linking to a parent chunk exceeds some threshold (say, cosine similarity), then “merge” smaller chunks into the bigger parent chunk.

The method will finally retrieve the parent chunk for better context.