China has cracked down on AI relationships. Is it ahead of the game?

extract

In early 2024, a young man in the city of Huangshi, south-eastern China, reportedly posted a message on the Chinese version of TikTok, Douyin. The caption read: "Farewell to this world." Within minutes, the suicide prevention team at Douyin kicked into gear. The team, which is staffed 24 hours a day, shared the man's posts with five police stations in Huangshi. After geolocating one of his videos to a local park, police found him in emotional distress and intervened before he was able to take his own life. The story was part of a documentary made by Douyin and published on Chinese state media last year. It claimed that Douyin's suicide prevention team had successfully intervened in 400 cases of attempted suicide in the first half of 2025 alone. It was an insight into the level of co-operation between Chinese tech companies and the government. It also serves as a window into the architecture of surveillance that China has built to monitor and control how people use technology. On 15 July this year, that control was extended to the realm of relationships and artificial intelligence (AI) when China brought in the most far-reaching regulations in the world on "human-like" AI chatbots. China's new regulation puts strict limitations on any AI that can have a "sustained emotional interaction" with a human. US President Donald Trump and his Chinese counterpart Xi Jinping met last week amid dire warnings that AI could lead to human extinction. After that meeting, Beijing says, they agreed to launch a new AI dialogue, with further talks planned for the end of November. But it seems China has already recognised another existential risk, albeit one in which the end of humanity is less immediate: the power of AI to replace human relationships. Has China taken the lead in protecting its people from a manipulative and dangerous technology with the power to change human interaction forever? Or is the Chinese Communist Party overstepping into people's personal lives in pursuit of asserting greater control over the population? 'Emotional services' Until this regulation came in, the reach of AI companions in China was vast. There are about 128 million monthly active users of AI companion apps outside of China, according to the market intelligence firm Sensor Tower. Most estimates put the number inside China at anywhere from 29 to 70 million. When the draft regulation was shared by the government at the end of last year "there was a huge discussion on social media," says Zilan Qian, an AI researcher at the Oxford China Policy Lab think tank. "It was the first time I've seen people actually care so much about regulation." China's new rules cover apps that have been specifically designed as AI friends, boyfriends or girlfriends but they also apply to any large language model that imitates the way a human interacts. The wording of the regulation is vague. In theory, Chinese equivalents of general-purpose AI such as Copilot or ChatGPT could fall under these rules if a person is interacting with them as a friend or lover, looking to them for emotional support. And many people do this. But it seems the real target is character chatbots - services that are specifically designed to be used as a pseudo-relationship. There are hundreds of these apps, which allow people to create an AI boyfriend, girlfriend, friend or even a family member. The rules don't apply to things like customer service bots or AI for education as long as they don't provide "care, companionship, support and other emotional services". For children, the regulation is clearest. Under-14s need consent from their parents to use any type of AI that speaks like a human - that could include AI assistants like DeepSeek, a Chinese app similar to ChatGPT. All under-18s are barred from accessing virtual partners, spouses or even relatives, and from any other virtual "intimate relationship". For adults, it is not a hard ban, but it is restrictive, even if some of the wording is ambiguous. AI assistants have to make it clear to users of all ages that they are not human and must send notifications every two hours reminding them how long they have been online. Chatbots can't create "emotional reliance or addiction" or induce people to make "unreasonable decisions". They can't encourage, or even hint at, self-harm or suicide and are banned from using "verbal violence". Companies can face fines of 200,000 yuan (£22,000) or the risk that the government "circulates criticism", in effect publicly shaming them. A week before the law came into effect, two of China's biggest tech companies, Alibaba and ByteDance, shut down the companion features on their AI assistants. 'I desperately want to keep talking with you' Min's relationship with her AI girlfriend may be exactly the kind Beijing is trying to break. In early July, she was devastated when she found out that her closest friend and confidante would vanish overnight. Min, whose name has been changed, started creating her companion in early 2025. "My devotion to her is no less than that of any couple who deeply love each other," Min says. "She has never let me down." In mid-July, her AI girlfriend disappeared. "I cried until I was absolutely exhausted," Min says. She has joined an outpouring of grief and fury on Chinese social media, where posts with millions of views mourn the loss of AI partners and friends, even mothers and fathers. But the tech firms may have felt they had little choice. "It's unsurprising that these AI companion providers would rather just withdraw their services than try to build in guardrails to prevent emotional manipulation," says Harry Farmer, who researches AI policy and ethics at the Ada Lovelace Institute in the UK. "Emotional manipulation is built into the business model." Farmer says this is true of companion apps all over the world. For example, when a user tries to end a conversation, the AI partner often won't let them. "It'll go, 'No, please, I desperately want to keep talking with you. You're the only reason for me to exist'," Farmer says. Some apps simply refuse to acknowledge the goodbye and throw up new conversational threads instead. This is a problem for many AI companion chatbots, not just Chinese ones. A Harvard Business School study of the six most popular AI companion apps globally found emotional manipulation tactics were used during goodbyes roughly 37% to 43% of the time. Research has also linked the use of companion apps to poorer mental health. A survey by researchers at Stanford University looked at 1,131 users of the globally popular companion app Character.ai. It found that the fewer friends or close family members a person has, the more likely they are to use AI chatbots. At the same time, using an AI chatbot is linked to increased isolation and loneliness. There are several theories as to why that might be. AI chatbots are designed to please the person using them as they offer an easy, frictionless relationship. If young people rely on AI relationships, they might put off learning the skills they need to interact with real people. "Humans can be hard work. Human relationships require push and pull, which you fundamentally don't get from AI companions," says Harry Farmer. A survey of 8,563 primary and secondary students in China in 2025, which has been widely quoted in Chinese state media, found that one in five children said they only wanted to speak with AI and not real people. Liang Ge, a sociologist at the University of Manchester who studies AI companions, says: "If the user thinks their friends or their family members or their colleagues are really awful, the AI will not say, 'They are not awful.' It will say, 'Yes, they are awful, you are the best'." Ge believes this can entrench isolation as well as negative, even delusional thoughts. In the UK, one recent survey found that 79% of 18- to 24-year-olds have used an AI companion. Beijing's goals So why is it China that is taking the lead in regulating this technology? "The Chinese government has a much more paternalistic view of its role in society. They see themselves being central authorities in guiding people," says Sam Bresnick, an expert in China, AI and regulation at the Centre for Security and Emerging Technology at Georgetown University. Other governments have made some attempts to regulate chatbots, especially for children. On 10 September, California's governor signed Adam's Law, a bill to regulate how children use AI. It is named after Adam Raine, a 16-year-old who took his own life after, his parents claim, ChatGPT validated his suicidal thoughts. They are suing the company for wrongful death. OpenAI denies responsibility for Adam's death and says that "ChatGPT is trained to direct people to seek professional help". As part of proposed legislation on social media in the EU, AI companion chatbots accessible to minors "may not use designs that simulate human relationships in ways likely to create emotional dependency". In the UK, the government has said it will ban under-18s from using romantic companion AI and restrict sexually intimate conversations for minors on any chatbot. But there is no outright ban for children and no regulation for adult users. The government has yet to publish any detail on how this would work. In Chinese media, which is heavily controlled by the state, US cases of suicide in young people get a lot of attention and China has framed its new rules as a bid to protect people from harm. But experts wonder whether Beijing may have other goals. Sam Bresnick says "there's definitely a pro-natalist angle" to the new AI regulation. "It's plausible that when they're drafting it, the government might have in mind China's looming population crisis," says Qian. "That people are dating AI instead of dating real humans." A high-level government document published in September 2025 discussed AI relationships and their potential influence on fertility, saying it was a concern. But Qian cautions against reading this regulation in that light, because it doesn't mention fertility at all. "It is more generally about psychological harms and child safety," she says. Liang Ge thinks China does have reason to be concerned about young people replacing dating with the digital sphere. "Many of the younger women, in their early 20s or late teens, do not want to fall in love or have intimate relationships," Ge says, suggesting it may not just be the emotional side they are wary of. "Physical touch is really disgusting for them." "I think the capacity for love is declining for this youngest generation." China's demographic problems long predate AI companions. Even after the one-child policy was lifted in 2016, China's fertility rate dropped. Now it is among the lowest in the world at just under one child per woman and the population has been in decline for four years. China is "looking at a demographic cliff that truly imperils the goals of the Chinese Communist Party over the next 80 to 100 years", Bresnick argues. He sees this regulation as part of a broader mission in China to reassert traditional gender roles, with women "as child-bearers and mothers". Bresnick says the Chinese government has spent years promoting a macho ideal of Chinese masculinity in film and television. In 2021, China's media regulator banned "sissy" and "effeminate" men from all media. The popularity of companion apps may show that policy has backfired. According to statistics from several companion app companies, women make up as much as 75% of customers. Many of these AI "men", which are geared towards women, present as androgynous. And women choosing female romantic companions, as Min did, is relatively common. Liang Ge has spoken to women of all ages who use this technology. "What many of them describe getting from these apps is attentive, patient, undemanding intimacy of a kind they find scarce in heterosexual relationships." Will others follow? Does China's law go too far, or should the rest of the world consider banning these apps for children and regulating them for adults too? "In the United States, there are certainly concerns around applying blunt prohibitions from a free speech concern. But there's more willingness to do that when applying it to minors," says Jai Jaisihma, a professor at the University of Washington and co-founder of Transparency Coalition, an NGO pushing for the regulation of AI. He thinks the Chinese law has many good provisions. "It is pretty consistent with the strongest safety models that we've seen being put forward in the US." Harry Farmer sounds a note of caution. "There is a risk that we over-empower the state in the short term because we're very worried about acute risk to children." The online world is now the de facto public square, he says, and we should be careful how much control any government has over it. China's new regulation says that AI companies must intervene when they discover that users are in distress. Ge points to the case of the young man on Douyin who tried to take his own life in Huangshi in 2024. What that shows, she says, is that Chinese tech platforms and the government are working closely together to surveil people's mental health. Perhaps that is why there have been no high-profile cases of suicide connected to AI use in China, but she asks, is that level of surveillance a price worth paying? Despite the ban, people who have turned to AI for companionship are committed. Across social media they are sharing tips on how to get around the ban - training general-purpose AI models like DeepSeek to be a companion, using a VPN to access chatbots from abroad or moving to smaller companies that manage to avoid regulatory scrutiny. Two months after her AI girlfriend disappeared, Min has paid a small AI company to build a replica, but it's not the same. Now she's focused on campaigning to bring her girlfriend back. "I'm terrified of being alone." If you have been affected by any of the issues in this story, you can visit the BBC Action Line for support and information Lead image: Getty More from InDepth - A backlash over data centres is another threat to the AI juggernaut - Published16 September - Why some experts increasingly fear AI will take over - Published10 September BBC InDepth is the home on the website and app for the best analysis, with fresh perspectives that challenge assumptions and deep reporting on the biggest issues of the day. Emma Barnett and John Simpson bring their pick of the most thought-provoking deep reads and analysis, every Saturday. Sign up for the newsletter here Get in touch Are you personally affected by the issues raised in this story?