A California bill that would regulate AI companion chatbots is close to becoming law

California has taken an enormous step towards regulating AI. SB 243 — a invoice that may regulate AI companion chatbots with a purpose to defend minors and weak customers — handed each the State Meeting and Senate with bipartisan help and now heads to Governor Gavin Newsom’s desk.

Newsom has till October 12 to both veto the invoice or signal it into legislation. If he indicators, it might take impact January 1, 2026, making California the primary state to require AI chatbot operators to implement security protocols for AI companions and maintain corporations legally accountable if their chatbots fail to fulfill these requirements.

The invoice particularly goals to forestall companion chatbots, which the laws defines as AI methods that present adaptive, human-like responses and are able to assembly a person’s social wants – from partaking in conversations round suicidal ideation, self-harm, or sexually specific content material. The invoice would require platforms to supply recurring alerts to customers  – each three hours for minors – reminding them that they’re talking to an AI chatbot, not an actual particular person, and that they need to take a break. It additionally establishes annual reporting and transparency necessities for AI corporations that supply companion chatbots, together with main gamers OpenAI, Character.AI, and Replika, which might go into impact July 1, 2027.

The California invoice would additionally enable people who imagine they’ve been injured by violations to file lawsuits in opposition to AI corporations in search of injunctive aid, damages (as much as $1,000 per violation), and lawyer’s charges. 

The invoice gained momentum within the California legislature following the death of teenager Adam Raine, who dedicated suicide after extended chats with OpenAI’s ChatGPT that concerned discussing and planning his demise and self-harm. The laws additionally responds to leaked internal documents that reportedly confirmed Meta’s chatbots had been allowed to have interaction in “romantic” and “sensual” chats with kids. 

In latest weeks, U.S. lawmakers and regulators have responded with intensified scrutiny of AI platforms’ safeguards to guard minors. The Federal Trade Commission is getting ready to research how AI chatbots impression kids’s psychological well being. Texas Lawyer Common Ken Paxton has launched investigations into Meta and Character.AI, accusing them of deceptive kids with psychological well being claims. In the meantime, each Sen. Josh Hawley (R-MO) and Sen. Ed Markey (D-MA) have launched separate probes into Meta. 

“I believe the hurt is probably nice, which suggests we’ve got to maneuver shortly,” Padilla informed TechCrunch. “We are able to put affordable safeguards in place to guarantee that significantly minors know they’re not speaking to an actual human being, that these platforms hyperlink individuals to the right assets when individuals say issues like they’re interested by hurting themselves or they’re in misery, [and] to verify there’s not inappropriate publicity to inappropriate materials.”

Padilla additionally burdened the significance of AI corporations sharing knowledge concerning the variety of occasions they refer customers to disaster providers annually, “so we’ve got a greater understanding of the frequency of this downside, quite than solely turning into conscious of it when somebody’s harmed or worse.”

SB 243 beforehand had stronger necessities, however many had been whittled down by amendments. For instance, the invoice initially would have required operators to forestall AI chatbots from utilizing “variable reward” ways or different options that encourage extreme engagement. These ways, utilized by AI companion corporations like Replika and Character, supply customers particular messages, recollections, storylines, or the flexibility to unlock uncommon responses or new personalities, creating what critics name a probably addictive reward loop. 

The present invoice additionally removes provisions that may have required operators to trace and report how typically chatbots initiated discussions of suicidal ideation or actions with customers. 

“I believe it strikes the appropriate stability of attending to the harms with out implementing one thing that’s both unattainable for corporations to adjust to, both as a result of it’s technically not possible or simply plenty of paperwork for nothing,” Becker informed TechCrunch. 

SB 243 is shifting towards turning into legislation at a time when Silicon Valley corporations are pouring millions of dollars into pro-AI political motion committees (PACs) to again candidates within the upcoming mid-term elections who favor a light-touch method to AI regulation. 

The invoice additionally comes as California weighs one other AI security invoice, SB 53, which might mandate complete transparency reporting necessities. OpenAI has written an open letter to Governor Newsom, asking him to desert that invoice in favor of much less stringent federal and worldwide frameworks. Main tech corporations like Meta, Google, and Amazon have additionally opposed SB 53. In distinction, solely Anthropic has said it supports SB 53. 

“I reject the premise that it is a zero sum state of affairs, that innovation and regulation are mutually unique,” Padilla mentioned. “Don’t inform me that we will’t stroll and chew gum. We are able to help innovation and growth that we predict is wholesome and has advantages – and there are advantages to this expertise, clearly – and on the similar time, we will present affordable safeguards for probably the most weak individuals.”

“We’re carefully monitoring the legislative and regulatory panorama, and we welcome working with regulators and lawmakers as they start to contemplate laws for this rising area,” a Character.AI spokesperson informed TechCrunch, noting that the startup already contains distinguished disclaimers all through the person chat expertise explaining that it ought to be handled as fiction.

A spokesperson for Meta declined to remark.

TechCrunch has reached out to OpenAI, Anthropic, and Replika for remark.