New California law forces chatbots to protect kids’ mental health

Gov. Gavin Newsom on Monday announced that he has signed Senate Bill 243, legislation that adds guardrails to AI-powered chatbots that operate in the state.

The legislation had divided tech industry representatives and child safety advocates. Newsom left unsigned another bill regulating such bots, Assembly Bill 1064, which child advocates argued better protected kids.

Under SB 243, companies that offer chatbots, such as OpenAI’s ChatGPT, would be required to institute specific safeguards. Among those would be requirements to monitor chats for signs of suicidal ideation, and to take steps to prevent users from harming themselves, such as by referring them to outside mental health assistance.

Makers of the chatbots would also be required to remind users that responses are artificially generated, and to create “reasonable measures” to prevent children from seeing sexually explicit content when using the bots. Kids using the bots would also get reminders to take breaks.

The legislation, among the first in the nation regulating chatbots, comes after a series of disturbing reports. Stories around the country have highlighted how the chatbots can seemingly feed delusions, or fail to pick up on signs of suicidal ideation. Meta, Facebook’s parent company, faced backlash this year after a leaked copy of its chatbot rules revealed the company allowed its bots to have “sensual” conversations with children.

SB 243 received support from a changing mix of backers, with tech industry group the Computer and Communications Industry Association ending up supporting it. After initially opposing the legislation, the group said after changes that it would “provide a safer environment for children, while also not creating an overbroad ban on AI products.”

But child safety advocates, after initially supporting SB 243, soured on the legislation after those changes, saying they had conceded too much to the tech industry. Two groups, Tech Oversight and Common Sense Media, instead threw their support behind AB 1064, saying the bill “establishes critical regulations for the development and use of artificial intelligence systems that interact with children.”

Under AB 1064, chatbots would not be allowed without tech companies showing they are “not foreseeably capable” of harming a child, such as by encouraging self-harm.

“We’ve seen some truly horrific and tragic examples of young people harmed by unregulated tech, and we won’t stand by while companies continue without necessary limits and accountability,” Newsom said in a statement announcing his signing of SB 243.

The statement did not mention any action on AB 1064. The governor has until the end of the day to veto or sign the bill, although it going unmentioned suggested his office was unlikely to move forward with that legislation.

This article was originally published on CalMatters and was republished under the Creative Commons Attribution-NonCommercial-NoDerivatives license.