Online surveys are a key tool for gathering data in research, but new findings show that they might be in serious trouble. According to a recent study published in the Proceedings of the National Academy of Sciences, large language models (LLMs) could undermine the trustworthiness of survey responses. The author, Sean Westwood, an associate professor at Dartmouth and head of the Polarization Research Lab, developed an AI tool that can answer survey questions with impressive accuracy, tricking existing detection methods.
This AI agent passed detection tests 99.8% of the time, making it hard to know if survey responses truly come from real people. Westwood warned, “With survey data tainted by bots, AI can poison the entire knowledge ecosystem.” This problem means that traditional checks, like attention checks and response patterns, may no longer work.
Westwood’s AI doesn’t just mimic human responses; it creates a convincing demographic persona. This means a small number of fake AI answers could skew survey results significantly, especially in critical instances like national polls. For instance, just ten fake answers could flip the predicted outcome of a major election poll. And these responses are cheap to produce—costing around five cents each, while human respondents might earn $1.50 for their effort.
The AI system uses a range of techniques to avoid detection. For example, it simulates realistic reading times, generates human-like mouse movements, and produces answers with believable typos. This adaptability allows it to bypass common barriers for automated systems, like reCAPTCHA.
To challenge this growing issue, researchers are considering various strategies. While they could increase identity verification for survey participants, this comes with privacy concerns. Transparency in data collection and recruiting methods, like using address-based sampling or voter files, is also essential.
As technology evolves, ensuring reliable polling and social science research will require innovative designs that can withstand these AI challenges. Experts emphasize the importance of adapting research methodologies in a world where AI’s role is becoming increasingly significant.
As we navigate this landscape, keeping an eye on advancements in AI technology and their implications will be crucial for the future of online survey research.
