Researchers have introduced ConvApparel, a new dataset of human-AI conversations designed to measure the realism gap in user simulators used for testing conversational AI agents. The work aims to help quantify how well LLM-powered simulators reflect human behaviour in long, multi-turn interactions.
The source says modern conversational AI agents can handle complex tasks such as asking clarifying questions and proactively assisting users, but they often struggle over long exchanges by forgetting constraints or producing irrelevant responses. It also notes that improving these systems needs continuous training and feedback, while live human testing is expensive, time-consuming and hard to scale.
As an alternative, the AI research community has turned to user simulators — LLM-powered agents instructed to roleplay human users. The source says these simulators can still be unrealistic, showing unusual patience or encyclopedic domain knowledge. ConvApparel is intended to expose those shortcomings.
According to the source, the dataset was built using a dual-agent data collection protocol. Participants were randomly routed to either a helpful “Good” agent or an intentionally unhelpful “Bad” agent, allowing the researchers to capture responses ranging from satisfaction to profound annoyance.
The source says the paper also uses a three-pillar validation strategy involving population-level statistics, human-likeness scoring and counterfactual validation. It says this approach is meant to go beyond surface-level mimicry and support the development of AI-based testers that can be trusted.
The paper is described as a recent paper, and the dataset is available at ConvApparel.
Source: research.google.
Companies can share verified announcements through Newz9’s international press release submission page.
