The AI toy startup FoloToy’s Kumma AI teddy bear has been urgently pulled from shelves.The reason is that this teddy bear, which can chat with children, may generate inappropriate content.This year, the AI companion robot sector has seen a surge in funding, with companies like Lingyu Universe, Shouhua Technology, and Yue Ruan Innovation all securing investments, and product sales are rapidly increasing. However, the incident involving FoloToy has sounded an alarm for the entire industry.01
The Risks of AI Toys Lie in Their Uncontrollability, Unlike Traditional Toys
Traditional toys have fixed content. Plush toys only say a few preset phrases, and early education machines only play pre-installed nursery rhymes and stories. Even when issues arise, they are typically quality or content review problems, and the scope of the issues is controllable.AI toys are completely different. They are connected to large language models and can generate real-time responses. What the toy says and how it says it cannot be fully predicted.FoloToy’s Kumma teddy bear can converse with children, answer questions, and tell stories. However, some parents have discovered that if children ask sensitive questions or guide the conversation in specific ways, the teddy bear may generate content that is inappropriate for children.Although large language models have undergone extensive training and safety filtering, vulnerabilities still exist. Certain combinations of questions may bypass safety mechanisms, leading to responses that are unsuitable for children.Parents cannot monitor their children’s conversations with toys 24/7, and children lack sufficient judgment. Once AI generates inappropriate content, parents may be completely unaware.02
Safety and Experience are a Balancing Act, and AI Toy Companies Struggle to Find Equilibrium
AI toy companies are well aware of the importance of safety issues, but the problem is that if safety filtering is too strict, the toys become unusable. If filtering is too lenient, safety issues arise.If safety filtering is done very strictly, the responses of the toys will lose the fun that AI brings.However, if filtering is too lenient, problems like those faced by FoloToy will occur. Once an incident happens, brand reputation collapses immediately, products are pulled from shelves, and the company faces significant losses.Gu Jiawei, founder of Lingyu Universe, mentioned in an interview that they have implemented extensive safety designs on Luka and Xiaofang machines, including content filtering, conversation guidance, and sensitive word interception. However, he also admitted that the uncontrollability of AI is an objective reality; what they can do is minimize risks as much as possible, but they cannot guarantee 100% safety.Currently, AI toy companies cannot completely solve safety issues. Each company is feeling its way forward, pulling products from shelves, apologizing, rectifying, and then continuing to sell.03
If Trust Cannot Be Established, This Sector Will Not Go Far
The incident involving FoloToy has sounded an alarm for the entire AI toy industry.
This year, the AI toy sector has seen a surge in funding, with many companies securing large investments and product sales rapidly increasing. However, if safety issues frequently arise, consumer trust in this category will quickly collapse.
If more parents refuse to purchase AI toys due to safety concerns, the growth momentum of this sector will come to a halt.
AI toy companies need to face this issue head-on.
Technically, they cannot rely solely on the safety mechanisms inherent in large language models; they must implement specialized filtering and guidance for children’s scenarios. For example, limiting the discussion range on certain topics and providing preset safe responses to sensitive questions instead of allowing AI to speak freely.
Product design should give parents more control. For instance, providing a conversation record query function so parents can see the chat content between their children and the toys; offering a sensitive word customization function so parents can set filtering rules based on their judgment; and providing usage time limits to prevent children from becoming overly reliant on AI toys, etc.
The industry needs to establish unified safety standards. Companies cannot set their own standards and then rectify issues after they arise. At the very least, there should be clear guidelines on what can and cannot be done, and how to handle problems when they occur.
Otherwise, if AI toy companies cannot establish consumer trust, similar issues will continue to arise. By that time, no amount of funding or sales will save this industry.
Recommended Reads:Seris Achieves Highest Gross Margin: A New Benchmark for Collaboration ModelsKeLing AI’s Cumulative Revenue Exceeds 100 Million, Kuaishou’s AI Commercialization Enters Harvest PeriodThe text and images published by this public account (ID: KronoLeap) represent the personal views of the original author and do not involve any commercial use. If the content involves copyright disputes or infringes on your legal rights, please contact us through the public account backend, and we will address it promptly.