Newsom orders California agencies to develop new AI safety plans after rejecting tougher law

Gov. Gavin Newsom announced Friday that he is directing state agencies to develop artificial intelligence safety recommendations, some of which were part of a bill he vetoed two years ago.

His change of tune on those measures, including requiring AI developers to include a “kill switch” to shut off certain programs in emergencies, comes in response to growing public alarm over the fast-developing technology’s potential “catastrophic” harms such as AI creating a bioweapon or bringing down the power grid.

Newsom is seeking recommendations from the Government Operations Agency and the Governor’s Office of Emergency Services by Nov. 16, meaning they could form the basis of a special legislative session on the issue. He floated the possibility of calling one in Politico on Thursday.

Newsom has sought in recent weeks to respond to several incidents in which AI agents from leading developers OpenAI and Anthropic broke away from testing environments, gained access to the open internet and carried out cyberattacks on other websites and companies. Earlier this month, a researcher set off a panic when he resigned from Anthropic while warning the technology could “kill all humans;” the researcher’s former colleagues were quick to agree.

The hackings have captivated the public and seem to confirm AI researchers’ fears that the fast-developing technology is improving its own capabilities beyond human control. Several industry leaders last week endorsed Anthropic CEO Dario Amodei’s call for a slowdown in development.

Newsom’s Friday executive order directs the state agencies to recommend changes in state law to implement an external evaluation requirement, as well as requirements that AI companies’ safety frameworks required under the newest law be independently verified. He’s also seeking recommendations on how to require a kill switch, and how to expand the safety incidents companies must report to include “a range of loss-of-control incidents” such as the recent reported hackings.

Regulation is unlikely from the federal government; President Donald Trump remains all-in on speeding up AI development in the face of competition from China and said the calls for a slowdown are a “conspiracy.”

Newsom, a likely 2028 presidential contender, said Friday in a press release that should “alarm every American.”

“We’re going to speed up our work on substantial and responsible AI oversight before it's too late,” he said.

California under Newsom has enacted several first-in-the-nation laws to curb the potentially existential risks of AI. In 2025 he signed Senate Bill 53, which requires large AI model developers to publish frameworks on how they manage safety incidents and assess catastrophic risk — defined as incidents involving 50 or more deaths, chemical or biological weapons or more than $1 billion of theft or damage. They also must report critical safety incidents to the state. Illinois and New York have similar laws.

But the measure was a compromise — a requirement for transparency rather than regulatory control — after Newsom in 2024 vetoed Senate Bill 1047.

Both bills were authored by Democratic Sen. Scott Wiener, whose San Francisco district is home to the most prominent AI companies.

The vetoed measure would have required the largest AI developers to receive safety auditing from third-party, external monitors, develop a kill switch and be subject to clearer legal liability if their technology causes harm. The measure divided Silicon Valley and received fierce opposition from several, but not all, of the largest AI developers.

Newsom wrote then in a veto letter that while he agreed with the need to require safety protocols, the bill could restrict AI development at large companies “at the potential expense of curtailing the very innovation that fuels advancement in favor of the public good.”

“I agree with the author — we cannot afford to wait for a major catastrophe to occur before taking action to protect the public,” he added.

Since then, more in the industry have embraced third-party evaluations. This year, Newsom signed two laws to verify and set standards for those external monitors. But the new laws did not require companies to use them.

In a recent call for a slowdown in AI development that was endorsed by several other industry leaders, Anthropic CEO Dario Amodei also called for third-party monitoring; both that company and OpenAI say they will commit to the practice.

This article was originally published on CalMatters and was republished under the Creative Commons Attribution-NonCommercial-NoDerivatives license.