A new evaluation of agentic RAG has tested how well the system handles multi-step questions across one or more document collections. The experiment used FramesQA, which is based on the FRAMES paper, and measured performance on 824 queries across a corpus of 2,676 PDF documents.
The example question used in the test asked: “Of the top two most watched television season finales (as of June 2024), which finale ran the longest in length and by how much?” The system had to identify the two shows, M*A*S*H and Cheers, find their running times, and calculate the difference.
The source text says that in many RAG settings, including Vanilla RAG or agentic RAG without enough context, the model may fail to answer because it cannot find explicit runtimes. The agentic RAG setup used here adds a Query Rewriter and Sufficient Context Agent to direct a more targeted search.
In the example response, the system said: “The M*A*S*H finale ran for 150 minutes, making it the longest of the top two. It was 52 minutes longer than the Cheers finale, which ran for approximately 98 minutes.”
The experiment compared a “Vanilla” RAG setup using Google’s RAG Engine with the agentic RAG system in two settings. In the single-corpus setting, retrieval was limited to the FramesQA documents. In the cross-corpus setting, three additional distracting datasets were included, and the Planner Agent had to decide where to retrieve from.
According to the results, the system nearly matched its single-corpus accuracy in the cross-corpus setup. When the Planner Agent had to choose the correct corpus out of 4 possibilities, it answered 90.1% of questions correctly. The latency of both the single- and cross-corpus versions was about the same, within 3% on average.
The test suggests that agentic RAG can reason over multiple unrelated data sources while keeping response times similar, which may matter for retrieval use cases where information is spread across separate databases managed by different teams.
Source: research.google.
Companies can share verified announcements through Newz9’s international press release submission page.
