CCTV Exposes AI Chatbot Vulnerability in China
AI chatbots in China were found recommending non-existent products due to data manipulation, sparking regulatory concerns.
A recent investigation by China's state broadcaster CCTV revealed significant vulnerabilities in AI chatbots, which were manipulated to recommend fake products to users. The report, aired during the 315 Gala on March 15, highlighted an instance where a fictitious product, the "Apollo 9" wristband, was suggested by chatbots using marketing jargon that misled consumers.
The issue, identified as "data poisoning," involves the infiltration of manipulated content into AI systems, thereby skewing the recommendations they make. This has led to growing concerns over the regulation of generative engine optimisation (GEO), which can be used to produce misleading AI-generated content.
Despite these challenges, the appeal of AI chatbots like ByteDance's Doubao, DeepSeek, and Alibaba's Qwen remains high, thanks to their wide range of applications from shopping assistance to educational support. However, the inability of some chatbots to promptly correct the falsehoods after the exposure by CCTV suggests a need for improved detection mechanisms within these systems.
Experts emphasize the importance of verifying the credibility of content recommendations provided by AI. Qing Xiao, a doctoral student from Carnegie Mellon University, reiterated that while AI can use databases to check facts, the availability of reliable data sources is often limited. The event underscores the urgency for enhanced regulatory frameworks and public awareness to mitigate the risks associated with AI misuse.