A test case comparing large language models on scientific literature found that models drawing from curated databases of experimental papers performed better than those trained on unfiltered internet data. The source text says NotebookLM and a custom-built tool outperformed LLMs relying on open web sources, which were more likely to mix established theories with highly speculative ones.
The evaluated LLMs, accessed in December 2024, also showed limits in temporal and contextual understanding. They often failed to detect when a hypothesis had later been disproved, and they frequently missed relevant papers when the user query did not match the exact wording used in the source material.
The results also point to a broader weakness in how LLMs handle tables and images, which are common in scientific papers. While two of the models consistently referenced images, the source text says they relied more on image captions than on visual analysis.
Improving visual reasoning, including the ability to interpret images, plots and scale bars, is identified as a major area for future work. For users and businesses that rely on AI to search scientific material, the findings suggest that source quality and visual understanding remain important limits.
Source: research.google.
Companies can share verified announcements through Newz9’s international press release submission page.
