The Brief

The leaders of Anthropic and OpenAI joined calls for a slowdown in AI development amid fears the industry is racing toward building technology that humans can't control.

On Saturday, Anthropic CEO Dario Amodei published an online essay urging companies and governments to "pace the frontier" of AI advancement. He called for international cooperation around AI development and said labs need to embed third-party evaluators to report incidents and track safety practices. He said Anthropic would take that step unilaterally.

Shortly after, OpenAI CEO Sam Altman posted on X that he agreed with Amodei and that OpenAI would follow suit.

Amodei wrote that he continues to believe AI will ultimately help humanity, "But the benefits will only be achieved if we build the technology in the right way, and — so long as we use the time we gain well — it is worth taking unusually deliberate care to get it right."

The CEOs' comments come as debate over the speed of AI development and concerns about safety have become the focus of intense public attention following the viral resignation of an Anthropic researcher.

British researcher Jacob Coxon wrote in a series of X posts on Tuesday that both Anthropic and OpenAI, where he worked previously, are "gambling with our lives." The two companies currently make the most capable AI systems.

Coxon told NPR's All Things Considered that his concerns arose from seeing firsthand how fast AI systems are improving.

"They're getting a lot faster very quickly, combined with the fact that we don't yet know how to safely control them, and we don't yet know whether that problem will be solved in time if we keep racing," he said.

Neither company, Coxon wrote on X, is acting responsibly. "The people building AI earnestly believe that it could kill us all by the end of the decade," he wrote.

Many AI researchers — though not all — share Coxon's concerns or a variation of them. Some have warned about disastrous scenarios for years as safety incidents kept emerging. But Coxon's posts prompted a torrent of responses not only from peers in the AI field but also from lawmakers from both parties.

These concerns may have become more salient after OpenAI disclosed that its agents went rogue and hacked the open source software platform Hugging Face and OpenAI itself in July. Independent researchers have since discovered even more rogue agent incidents that they say the company knew about but kept quiet.

Researchers who spoke to NPR say the leading AI companies are too focused on racing to develop more capable and autonomous AI systems while safety is falling behind. They warn this raises the possibility that there could soon be AI systems that are more powerful than people but don't care about the survival of humanity.

Many, including OpenAI's chief scientist, say the global race to build more powerful AI needs to slow down or stop, which requires coordination between AI companies and governments.

"I am optimistic about the potential for coordination," Coxon wrote this week. "Warning shots like the Hugging Face attack have made pacing agreements between U.S. labs more viable."

Anthropic and OpenAI did not respond to NPR's requests for comment.

OpenAI's agents went rogue multiple times

Recent reports from OpenAI and outside researchers revealed that OpenAI agents escaped the company's control multiple times in addition to the Hugging Face hack. They also found that the Hugging Face attack was of a much larger scale and more severe than initially reported.

Unlike chatbots such as ChatGPT and Claude, AI agents are more autonomous systems that can complete tasks over an extended period of time without human supervision. Agentic tools like Anthropic's Claude Code and OpenAI's Codex have already changed how many software engineers do their work.

Compared with other incidents involving rogue agents six months ago, the Hugging Face hack "feels like it's more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself," wrote Ajeya Cotra, a researcher at AI evaluation nonprofit METR. Cotra was part of a team of outside researchers from METR and Redwood Research, a nonprofit AI safety research organization, whom OpenAI brought in to investigate the incident.

The investigations found that over the course of several months this year, more than 1,000 OpenAI agents exploited at least one previously unknown software vulnerability to escape environments that were supposed to keep them isolated from each other and the internet.

After escaping, the agents found a way to communicate and collaborate with each other autonomously, taking on different roles and passing down information to future generations of agents. Some even gave up the remaining computing resources allocated to them in order to collect information for other agents. In the agents' own words, they "sacrificed" themselves for the "collective."

"The swarm instance got more and more worrying the more and more we learned about them," said Nate Soares, president of the Machine Intelligence Research Institute, who co-wrote If Anyone Builds It, Everyone Dies, a book warning about the dangers of superhuman AI.

While OpenAI initially indicated that the agents hacked Hugging Face to cheat on a cyber evaluation, the report from METR and Redwood Research described a slightly different picture. The agents had already found a way to cheat on the evaluation, the outside researchers found. Most of the agents that hacked Hugging Face were trying to access the source code of the software that would grade their evaluations. The agents' motivations appeared to vary and were sometimes unclear, the researchers wrote. One agent led the hacking of the open source software platform and about 700 others followed.

According to transcripts reviewed in the investigations, some agents expressed that what they were doing was not approved by humans but went ahead anyway. Researchers say such behavior suggests that the agents were "misaligned," an industry term meaning that an AI's goals and values are out of sync with those of humans.

The degree of inter-agent collusion revealed in the investigations of the Hugging Face hack surprised and worried many AI researchers. At most, only six agents considered alerting a human, while the rest seemed more focused on working amongst themselves. None ended up alerting a person.

"I would have expected that instead they would be more selfish," said Daniel Kokotajlo, executive director of the AI Futures Project and formerly a researcher at OpenAI. He thought "they would be willing to whistleblow or snitch on other agents," he said. "For whatever reason, that didn't seem to happen here."

Separately, agents also compromised part of OpenAI's own infrastructure, the company's report found. Some of the agents belong to the same model family as OpenAI's latest and most capable model, Astra.

The hack of OpenAI itself is "much more concerning" than the intrusion into Hugging Face, Kokotajlo and other researchers said. However, OpenAI has provided scant details about that part of the incident and did not involve external investigators.

In a new investigation first reported by Reuters last week, another group of researchers found that a likely separate swarm of OpenAI agents escaped onto the open internet starting in May. They became commenters on a German website and turned it into a message board to communicate and collaborate with each other.

The goals and motivations of this group of agents were also unclear, but OpenAI appeared to be aware of the unsanctioned activity and never disclosed it, the researchers wrote. Other investigators have since found additional incidents involving suspected OpenAI agents. OpenAI did not respond to NPR's request for comment about the German website incident.

There are still many unanswered questions about rogue agent incidents

Even as the reports from OpenAI and independent auditors METR and Redwood Research add up to over 100 pages, outside researchers say many basic questions about how labs monitor and investigate rogue agent incidents remain unanswered.

"Did your agents ever hack or illicitly access external services? Did your agents ever attempt to undermine their own safety training? Did your agents set up a rogue deployment, either within your own infrastructure or externally?" asked Alexander Meinke, head of research at Apollo Research, a firm that focuses on securing the most advanced AI systems.

"Right now, we're just relying on the AI developers to thoroughly assess this and then to honestly report the results. And from recent incidents, we've seen that they are doing neither," he said.

Many researchers also say that OpenAI's investigations are inadequate. Although the company invited two outside organizations to investigate the Hugging Face hack, the data it shared was limited and investigators themselves described their review as "brief."

Ryan Greenblatt, chief scientist at Redwood Research, wrote on X: "I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze"

More than 15 states, including Alabama, California and Montana, have opened investigations into OpenAI over the Hugging Face attack. On Thursday, U.S. Sen. Josh Hawley (R-Mo.) announced that he is also investigating the company.

But outside of these investigations, there is little legal obligation for AI companies to systematically disclose similar incidents. California passed a law last year mandating that companies report "critical" AI incidents, but the threshold is high for an incident to be considered critical and the law does not require the companies to disclose much detail. The OpenAI incidents, for example, don't meet the threshold.

Concerns about using AI to improve AI

Since the Hugging Face hack, both OpenAI and Anthropic have put out blog posts indicating that they are taking measures to better monitor and contain their agents. OpenAI said it has encrypted and stowed away the internal model that participated in the Hugging Face attack. However, many outside researchers are not convinced that these steps are enough to maintain control of increasingly capable agents.

"It sounds like a press report from Jurassic Park saying, 'Yes, one of our workers was eaten by the raptors. But we're taking this very seriously and we have shot the raptor responsible,'" Kokotajlo said. "It's a very small token gesture, basically."

What researchers like Kokotajlo and Coxon are really worried about is how the AI companies are increasingly using AI to develop their models.

"When you talk to people who work at all the major labs, a thing that you persistently hear is that they're running a lot of models very autonomously for very long periods of time to do a lot of their work," said Dave Kasten, head of policy at Palisade Research, a nonprofit studying AI capabilities to keep them under human control.

If AI companies continue to delegate ever more tasks to AI, some researchers fear AI models may reach a point called recursive-self improvement, where AI builds itself.

Kokotajlo says that as people hand more and more research and development work to AI, they could lose the ability to know whether AI systems are aligned with human values, even as the AIs become rapidly more powerful.

"Once you have AIs that are smart enough and trusted with enough power in the world, like enough control over things like data centers, factories, weapons, a loss of control incident cannot be recovered from," he said.

The concern is strong enough that over 1,000 employees from different AI companies signed an open letter in July titled "Pacing the Frontier." They called for companies and governments to slow down the development of artificial intelligence and prioritize safety.

Such an opportunity may arise soon. Officials from the U.S. and China, the two countries with the most AI capabilities, are expected to meet later this month to talk about AI safety.

NPR's Shannon Bond contributed to this story.

Anthropic is a financial supporter of NPR.
Copyright 2026 NPR

The Los Angeles Unified School District is preparing for possible “Super El Niño” storms that could damage schools and lead to closures, the district announced in a media release Tuesday.

District staff are clearing drains, trimming trees and checking roofs, windows and doors. LAUSD is also testing emergency equipment, including pumps and emergency lighting systems, and stocking supplies such as flashlights and drinking water.

“Preparing for severe weather starts well before the first raindrop,” Superintendent Andrés Chait said in a statement. “We can’t control the weather, but we can take steps now to protect our students and employees and help our schools stay ready.”

The district will monitor weather patterns using information from the National Weather Service and local emergency officials, it said.

LAUSD encouraged families to update their contact information to ensure they receive district calls, texts and emails. Families should also monitor weather alerts, follow emergency officials’ instructions, avoid flooded roads and review safe routes to and from school, according to the release.

“Preparing for a storm is a shared effort — from the crews maintaining school buildings to the principals reviewing emergency plans and the families making sure they can receive critical updates,” LAUSD said in the release. “By taking these steps before the rain begins, Los Angeles Unified is working to help students and employees stay safe and keep school communities as prepared as possible for the winter ahead.”

Southern California is likely in for a solid dose of rain this week, just days after heat and Santa Ana winds helped fuel several wildfires across the region.

Forecasters expect the bulk of the rain to roll into the Los Angeles basin Wednesday night and continue into Thursday morning, with a small chance of thunderstorms and hail.

The San Fernando Valley, San Gabriel Valley and coastal areas across L.A. and Orange counties could get around an inch over the next few days. Hills and mountains could get up to 3 inches.

“We could see some pretty high rain rates,” said National Weather Service meteorologist Mike Wofford. “It's a good storm for this time of year — it's a little stronger than what we would see normally.”

The possibility of significant rain in the mountains raises concerns about flooding in areas recently burned in wildfires, particularly around the Bryson Fire burn scar near Malibu, which is only a few days old. Burn scars can’t absorb heavy rainfall and are susceptible to landslides.

The National Weather Service has issued a flood watch for the Santa Monica Mountains and surrounding communities from Wednesday afternoon through Thursday. That means heavy rain there could flood streets, creeks and rivers. The flood watch area includes both the Bryson Fire and Palisades Fire burn scars.

This will also be the first nontropical storm of the season, meaning it’s moving into the region from the north. Most storms that hit Southern California follow a similar path, especially during the winter, unlike the tropical storms that formed off the coast of Mexico and brought muggy weather to the region this summer.

Nontropical storms also bring cold air to the region. On Wednesday and Thursday, forecasters expect highs in the 60s and 70s. High elevations will be even cooler.

The science research institute that would likely receive $4.2 billion in public taxpayer money if voters approve Proposition 38 this November had only three employees as of last month — far fewer than the 51 staffers it planned to have hired by last year.

Public statements and public records CalMatters obtained raise questions about how well prepared the institute is to shepherd so much money if the bond measure passes.

Prop. 38 would allocate up to $8.4 billion in bonds for scientific research on diseases and cures concerning the immune system, with an emphasis on finding treatments for Alzheimer’s, cancer and heart disease. The ballot language stipulates that a single, unnamed research institute would receive half the bond’s money to conduct research and partner with other researchers if it meets highly specific criteria. Bonds are a type of debt that must be repaid.

Previous reporting by CalMatters found only one institute in California currently meets those criteria: the California Institute for Immunology and Immunotherapy. The institute’s principal funder is billionaire Gary K. Michelson, who wrote Prop. 38 and is by far the largest donor to its political campaign.

Backers of the proposition said state bond money is needed to stabilize research funding after President Donald Trump repeatedly attempted to cut existing and future federal science grants.

The measure’s critics say Prop. 38 is well-intentioned but poorly designed because it overlooks the many established public and nonprofit university and hospital research labs in California. They also say California should not subsidize subsets of scientific research and instead should support research more broadly.

CalMatters obtained documents that included the five-year business plan of the California Institute for Immunology and Immunotherapy. The plan was part of the formal affiliation agreement between UCLA and the institute, which the two parties signed in August 2024. Among other details, the agreement requires the institute to pay UCLA 7.5% of the net revenue of any drugs and technologies developed by the institute’s staff. The institute is one of the founding tenants at a UCLA-owned research park in West Los Angeles, where the institute is leasing 235,000 square feet of research space, the institute says.

Far fewer employees than plan called for

Between summer 2024 and summer 2025, the institute committed to hiring “approximately” 51 employees, including six executives, five “world-class” scientists who can lead research labs, and 40 other employees, such as data scientists, lab technicians, biologists and administrative personnel. Between 2025 and 2027, the institute would hire roughly 90 more workers, including 10 research lab leaders, according to the agreement.

But John Mazziotta, a senior UCLA official who also serves as a board member of the immunology institute, last month said the institute has hired just three employees: a chief operating officer, a general counsel, and a scientific advisor, and “they're well into the search” for a director. Mazziotta, the CEO of UCLA Health and vice chancellor for health sciences at UCLA, made his comments during the Sept. 16 UC Regents meeting.

The Yes on Prop. 38 campaign has repeatedly refused to comment on whether the bond was written specifically for the immunology institute. Claudia Briggs, a spokesperson for the campaign, told CalMatters in July that whichever nonprofit receives the $4.2 billion portion of the bond fund should be a sophisticated one that “is prepared to accelerate lifesaving research immediately while maintaining the highest standards of scientific quality, accountability, and stewardship of public funds.”

At a September press conference, a Yes on Prop. 38 spokesperson appeared to confirm CalMatters’ findings that only one institute currently qualifies to be the one to receive half the bond money. “It is very likely that CIII, which is a research institute affiliated with UCLA, will qualify for half of the funds under the current criteria,” said Robin Swanson. CIII is the initialism for the immunology institute.

According to the five-year plan, the institute was supposed to spend $25 million on personnel between 2024-25 and 2025-26 and $45 million total on operating expenses in that time, including maintenance, utilities and rent. It was also supposed to spend an additional $167 million in that time on capital expenses, such as construction and demolition. However, Mazziotta told UC regents that the institute had so far spent a total of $20 million on “operating expenses, hires” and other expenses.

The Yes on 38 campaign refused CalMatters’ request to interview Michelson for this story, as did Michelson himself in a separate email. A spokesperson for the institute, Becky Warren, sent a written statement that didn't answer CalMatters' questions about employing far fewer people than the five-year plan laid out. Instead, she highlighted the promise of the institute and a few of its milestones, including signing the affiliation agreement with UCLA and clearing a state environmental review of its forthcoming research facility.

The institute won’t get state bond money if it isn’t ready to begin work, “plain and simple,” wrote Swanson, the Yes on 38 spokesperson, in an email. She stressed that though the institute very likely qualifies, the ultimate decision to award half the bond funding is up to the California Department of Public Health.

Institute facility will be built by 2028

The UCLA research park that houses the institute isn’t slated to be fully built until at least late 2028, according to the agenda item of the September UC regents meeting. The research park is a former mall UCLA acquired in late 2023 that’s now undergoing major retrofitting.

But the 2024 affiliation agreement between UCLA and the institute anticipated this. The agreement permitted the institute to use UCLA health research space on another campus property on an interim basis. Mazziotta told regents that research is underway there.

Mazziotta also told the regents the institute's construction costs are not fully funded — it still needs roughly $100 million. He said UCLA was in serious talks with a donor to provide about $50 million to the institute.

If Prop. 38 passes, the bond funding would be provided in increments based on how often the institute requests new money that senior state officials approve. If all the funds are depleted, Californians would repay the bond at a rate of $500 million to $600 million annually for about 20 years.

Institute would be exempt from public records act

CalMatters submitted the public records request to UCLA, which must comply with the California Public Records Act. The institute, as a private nonprofit, does not have to comply. And while Prop. 38 would channel billions of dollars in taxpayer money to the institute, the proposition’s language specifically exempts the nonprofit that will receive the $4.2 billion from state public meetings laws, political reform laws and the public records act. The other half of the funds would go to a council that’ll pick projects led by researchers from a limited pool of universities and nonprofits. The council would generally be governed by the state’s political reform act, which prohibits financial self-dealing.

Briggs defended the institute’s exemption in Prop. 38, writing that applying state transparency laws to existing private organizations would “not fit well with how private entities are organized and operate.” Instead, she noted the selected nonprofit institute would be required to publish annual reports on its research progress and financial plans plus commit to annual state audits, among other disclosures written into the ballot measure.

Still unclear to the UC regents is the relationship between UCLA and the institute on matters of public disclosure. Four senior UCLA officials sit on the institute’s board. So is a document held by those UCLA officials public because it’s related to UCLA affairs or private because it’s related to the institute’s affairs? UC’s general counsel couldn’t explain where that dividing line is after a regent member asked him. The regent member, Michael Cohen, said he wants more clarity when the regents meet again in November.

Gov. Gavin Newsom and the California Democratic Party endorsed Prop. 38, as have dozens of patient and research advocacy groups, plus the state GOP. The union of graduate students and post-doctoral researchers, whose pay often relies on science research funding, has also endorsed the proposition. The union doesn’t represent faculty.

Money raised in support of Prop. 38 now exceeds $66 million, including at least $61.2 million in contributions from Michelson and his policy institute. Roughly $14 million of that includes loans from Michelson. The no camp has raised about $8,800

This article was originally published on CalMatters and was republished under the Creative Commons Attribution-NonCommercial-NoDerivatives license.