The Brief

California government agencies are going all-in on generative artificial intelligence tools after Gov. Gavin Newsom’s 2023 executive order to improve government efficiency with AI.

One deployment recently touted by the governor is a chatbot from the California Department of Forestry and Fire Protection, the primary agency tasked with coordinating the state’s wildfire response.

The chatbot, which Cal Fire says is independent of Newsom's order, is meant to give Californians better access to “critical fire prevention resources and near-real-time emergency information,” according to a May release from Newsom’s office. But CalMatters found that it fails to accurately describe the containment of a given wildfire, doesn’t reliably provide information such as a list for evacuation supplies and can’t tell users about evacuation orders.

Newsom has announced AI applications for traffic, housing and customer service to be implemented in the coming months and years. But Cal Fire’s chatbot issues raise questions about whether agencies are following best practices.

“Evaluation is not an afterthought,” said Daniel Ho, law professor at Stanford University whose research focuses on government use of AI. “It should be part of the standard expectation when we pilot and roll out a system like this.”

The chatbot uses the Cal Fire website and the agency’s ReadyForWildfire.org to generate answers. It can tell users about topics such as active wildfires, the agency, fire preparedness tips and Cal Fire’s programs. It was built by Citibot, a South Carolina-based company that sells AI-powered chatbots for local government agencies across the country. Cal Fire plans to host the tool until at least 2027, according to procurement records.

“It really was started with the intent and the goal of having a better-informed public about Cal Fire,” said Issac Sanchez, deputy chief of communications for the agency.

When CalMatters asked Cal Fire’s bot questions about what fires were currently active and basic information about the agency, it returned accurate answers. But for other information, CalMatters found that the chatbot can give different answers when the wording of the query changes slightly, even if the meaning of the question remains the same.

For example, an important way Californians can prepare for fire season is assembling a bag of emergency supplies should they need to evacuate. Only “What should I have in my evacuation kit?” returned a specific list of items from Cal Fire’s chatbot. Variations of the question that included “go bag,” “wildfire ready kit” and “fire preparedness kit” instead returned either a prompt to visit Cal Fire’s “Ready for Wildfire” site, which has that information, or a message saying “I’m not sure about the specific items you should have” and the wildfire site link. Two of those terms are present on the site the chatbot referenced.

And while the chatbot didn’t generate incorrect answers in any of the queries CalMatters made, it doesn’t always pull the most up-to-date information.

When asked if the Ranch Fire, a 4,293-acre fire in San Bernardino County, was contained, the chatbot said that the “latest” update as of June 10 showed the fire was 50% contained. At the time CalMatters queried the chatbot, the information was six days out of date – the fire was 85% contained by then.

Similarly, when asked about current job openings at the agency, the chatbot said there weren’t any. A search on the state’s job site showed two positions at Cal Fire accepting applications at the time.

Mila Gascó-Hernandez is research director for the University at Albany’s Center for Technology in government and has studied how public agencies use AI-powered chatbots. Two key factors she uses to evaluate such chatbots are the accuracy of information they provide and how consistently they answer the same questions even if the question is asked in different ways.

“If a fire is coming and you need to know how to react to it, you do need both accuracy and consistency in the answer,” she said. “You’re not going to think about ‘what’s the nice way to ask the chatbot?’”

Currently, the chatbot is unable to provide information about evacuation orders associated with fires. When asked who issues evacuation orders, it sometimes correctly said law enforcement, while other times said it didn’t know. Cal Fire’s Sanchez said it’s reasonable to expect the chatbot to be able to answer questions about evacuations.

If there are no evacuation orders for a particular fire, he said, “the answer should be ‘there doesn’t appear to be any evacuations associated with this incident.’”

Sanchez said he and his team of about four people tested the chatbot before it went out by submitting questions they expected the public to ask. Cal Fire is currently making improvements to the bot’s answers by combing through the queries people make and ensuring that the chatbot correctly surfaces the needed answer.

When CalMatters asked the bot “What can you help me with?” in early May, it responded, “Sorry I don’t have the answer to that question right now” and asked if CalMatters had questions about information on the Cal Fire site. By mid-June, that answer was updated to being able to “provide answers to questions related to information located on this page such as details about current fires, CAL FIRE job classifications, examination requirements and CAL FIRE's various programs.”

“The big message we want to get across,” Sanchez said, “is be patient.”

But experts said the process of kicking the tires on a chatbot should happen long before procurement begins.

The preferred process, Stanford’s Ho said, is to establish criteria for how the chatbot should perform before a vendor is selected so there are clear benchmarks to evaluate the tool. Ideally, those benchmarks are created by an independent third party. There should also be an evaluation of the benefits and risks before the chatbot is released.

And in a best-case scenario, the public would be involved before launch, Albany’s Gascó-Hernandez said. Agencies interested in using chatbots should identify the questions the public is likely to ask the AI tool ahead of time, ensure those are representative of the expected population the agency serves and refine the chatbot by having members of the public pilot the system to ensure it provides the kind of information they seek.

“These user engagement and user experiences are very important so the citizen ends up using the chabot,” she said.

This article was originally published on CalMatters and was republished under the Creative Commons Attribution-NonCommercial-NoDerivatives license.

The U.S. Department of Housing and Urban Development cannot impose dramatically different conditions for homelessness programs for now, according to an oral ruling Friday by U.S. District Judge Mary McElroy in Rhode Island.

McElroy granted a preliminary injunction to a group of states, cities and nonprofits who said a last minute overhaul of how to spend $4 billion on homelessness programs was unlawful. She also agreed with their argument that it likely would push many people back onto the streets in the middle of winter, causing irreparable harm.

"Continuity of housing and stability for vulnerable populations is clearly in the public interest," said McElroy, ordering HUD to maintain its previous funding formula.

The National Alliance to End Homelessness, one of the plaintiffs, said in a statement the order "means that more than 170,000 people – families, seniors, veterans, and people with disabilities — have respite from the government's assault."

HUD has sought to dramatically slash funding for permanent housing and encourage more transitional housing that mandates work and treatment for addiction or mental illness. The overhaul — announced last month — also would allow the agency to deny money to local groups that don't comply with the Trump administration's agenda on things like DEI, the restriction of transgender rights and immigration enforcement.

"HUD will continue working to provide homelessness assistance funding to grantees nationwide," said HUD spokeswoman Kasey Lovett in a statement to NPR. "The Department remains committed to program reforms intended to assist our nation's most vulnerable citizens and will continue to do so in accordance with the law."

'Chaos seems to be the point'

McElroy expressed frustration with a series of HUD actions in recent weeks. Just hours before a Dec. 8 hearing, the agency withdrew its new funding notice, saying it would make changes to address critics' concerns. But on Friday, HUD's attorney said the new version would not be ready until the end of the day.

"The timing seems to be strategic," McElroy said, asserting there was no reason the document could not have been ready before the hearing. "The constant churn and chaos seems to be the point."

In defending the agency, attorney John Bailey said HUD was simply trying to change its policies to reflect President Donald Trump's executive orders, which he called "legal directives." The judge interjected repeatedly to explain that he was conflating things, noting Congress — not the president — makes laws.

'It's kind of shocking'

HUD's changes were announced in November with little notice and only weeks before local homeless service providers must apply for new funding.

"Our agencies are just scrambling right now to try to respond," said Pam Johnson with Minnesota Community Action Partnership, whose members provide housing and other services for homeless people. "It also just reverses 40 years of bipartisan work on proven solutions to homelessness. So it's really, it's kind of shocking."

For decades, U.S. policy favored permanent housing with optional treatment for addiction or mental illness Years of research has found the strategy is effective at keeping people off the streets.

But many conservatives argue it's failed to stop record rates of homelessness.

"What is the root cause of homelessness? Mental illness, drug addiction, drug abuse," HUD Secretary Scottt Turner said recently on Fox Business Network. "During the Biden administration, it was just warehousing. It was a homeless industrial complex."

Turner and others who support the changes say the goal is to push people towards self-sufficiency.

But local advocates say mental health and substance abuse are not the main factors driving homelessness.

"It's poverty. Poverty, low income and significant lack of affordable housing," says Julie Embree, who heads the Toledo Lucas County Homelessness Board in Ohio.

Many in permanent housing have disabilities that make it hard to work full time, she said. Embree agrees with Trump administration goals like efficiency and saving money, but says pushing people back into homelessness, where they're more likely to land in jail, the courts or a hospital, is not cost-effective.

"One emergency room visit is just as expensive as a month of sustaining this [permanent housing] program," she said.

In Los Angeles, Stephanie Klasky-Gamer with LA Family Housing said there is a need for more transitional housing, but not at the expense of long-term housing. And the idea that programs could simply switch from one to the other is not only unrealistic, it's illegal.

"You cannot take a building that has a 75-year deed restriction and just — ding! — call it interim housing," she said.

Those challenging HUD say providers who own such properties – or states who've invested millions of dollars in permanent housing projects — face "significant financial jeopardy" if their funding is not renewed.

In addition to the legal challenges, members of Congress from both parties have questioned HUD's sudden shift on homelessness. Advocates have lobbied lawmakers to step in and, at the least, push for more time to prepare for such a massive overhaul.

President Donald Trump said the administration has reached agreements with nine more drugmakers to bring their U.S. drug prices more in line with what other wealthy countries pay.

Fourteen companies in total have now reached what the administration calls most-favored-nation pricing deals. The companies that took part in Friday's announcement were: Amgen, Boehringer Ingelheim, Bristol Myers Squibb, Genentech, Gilead Sciences, GSK, Merck, Novartis and Sanofi.

They agreed to charge the U.S. government no more for new drugs than the prices paid by other well-off countries. The agreements will allow state Medicaid programs to access lower prices from the nine companies. In a statement, the White House said the change will result "in billions of dollars in savings."

The drugmakers also agreed to invest at least $150 billion in manufacturing operations in the U.S. The president is seeking to increase domestic production of pharmaceuticals.

In addition, the companies agreed to make some of their most popular drugs available at lower prices to consumers who pay out of pocket through a government website called TrumpRx.com. The TrumpRx website is expected to launch in early 2026, and would take consumers to pharmaceutical companies' direct-to-consumer websites to fulfill orders.

For example, Merck will reduce the price of Januvia, a medication for Type 2 diabetes, from $330 to $100 for patients purchasing directly through TrumpRx, the White House said. Amgen will reduce the price of Repatha, a cholesterol-lowering drug, from $573 to $239 when purchased through TrumpRx.

In exchange for these concessions, the companies will be exempt from possible administration tariffs for three years.

The extent of savings for consumers under the agreements is unclear. Medicaid and its beneficiaries already pay some of the lowest prices for drugs. And people with health insurance could spend less on copays for their medicines than paying cash for them through the drugmakers.

Separately, Trump said during the press event that he would like to get health insurers to lower their prices, too.

"I'm going to call a meeting of the insurance companies," he said. "I'm going to see if they [will] get their price down, to put it very bluntly."

America's new top health diplomat is Bethany Kozma.

The job she took on this week — leading the Department of Health and Human Services Office of Global Affairs — does not have a high profile. And Kozma herself is not a familiar name in the world of public health.

But it is a position with power — and Kozma has a record of public statements and activism on health issues, equating abortion with "murder" and campaigning against gender-affirming care.

The office sometimes is referred to as the "diplomatic voice" of HHS. As director, Kozma will have considerable influence over how the U.S. shapes health policy in other countries in the wake of the Trump administration's foreign aid cuts and withdrawal from the World Health Organization.

Kozma declined to be interviewed for this story. She doesn't appear to have a background in global health based on publicly available information online. The HHS website offers few details about her professional profile. In response to questions about her qualifications and vision for the role, HHS responded with this statement.

"The Office of Global Affairs (OGA) advances the Trump administration's agenda and priorities by bringing common sense, transparency and gold-standard science to global partners. Under Secretary Kennedy's leadership, OGA is committed to strengthening the United States' position as the global gold-standard for public health and ensuring Americans are protected at home and abroad."

Who is Bethany Kozma?

Kozma began her career in public service during the George W. Bush administration, working at the White House Homeland Security Council. During the Obama years, she re-entered public life as an activist.

In a 2016 commentary for The Daily Signal, a conservative news website founded by the Heritage Foundation, she argued against the Obama administration's guidance that public schools should allow children to use the bathroom that comports with their identity.

"This radical agenda of subjective 'gender fluidity' and unrestricted shower and bathroom access actually endangers all," she stated, noting that "predators" could abuse the policy.

In 2017, she joined the Trump administration as senior adviser for Gender Equality and Women's Empowerment in the United States Agency for International Development, eventually being promoted to deputy chief of staff. In videos obtained and released by ProPublica, Kozma recalls calling the U.S. a "pro-life" country in a closed-door U.N. meeting about women's rights in 2018, when access to abortion still was protected nationally by Roe v. Wade.

In August 2020, Sen. Chris Coons, D-Del., and four other Democratic senators issued a letter labeling Kozma and several other political appointees at USAID as "prejudiced" and called for them to be removed from their posts. Kozma has "spoken extensively and derisively of trans people and trans issues," the senators wrote.

During the Biden administration, she also was involved in Project 2025, the Heritage Foundation's "blueprint" for a new Republican administration. She played a prominent role in Project 2025 training videos, obtained and published by ProPublica.

In one nearly 50-minute training video focused on left-wing language, she called for a Republican administration to "eradicate 'climate change' references from absolutely everywhere," and said that concerns over climate change are efforts at "population control." She also called gender-affirming care "absolutely infuriating" and said "the idea that gender is fluid is evil." Overall, she argued that changing language around these policies should be a priority for political appointees.

Kozma joined the second Trump administration as a chief adviser at the HHS Office of Global Affairs. In September, she spoke at a U.N. event commemorating the 30th anniversary of the declaration that denying women's rights is a human rights violation.

"While many may celebrate so-called successes gained for women over the last 30 years, one must ask what defines true success for women?" she began, adding that "biological reality is rooted in scientific truth and is confirmed by the universal truths that we are endowed by our creator who made us 'male and female.'"

Those views can be divisive but have garnered some support for Kozma's promotion.

"Bethany is an excellent pick for global affairs at HHS," says Roger Servino, vice president of domestic policy at The Heritage Foundation. "She was an early champion of protecting children from gender ideology back when the medical establishment was able to silence voices of reason and dissent and she is perfectly placed to help push back on global health bodies trying to impose left wing pseudoscience on the American people and the world."

What will her goals be at the Office of Global Affairs?

Kozma is taking over as director of the HHS Office of Global Affairs at a time of drastic change for global health.

In previous administrations, a main focus of the office was dealing with the World Health Organization. Typically, the director, who usually has a background in public health, is involved in negotiations on sharing data for pathogen surveillance or developing vaccine policy, for example.

After President Trump withdrew the U.S. from WHO, the administration has started a new strategy: striking deals with individual countries to give health aid in exchange for their meeting certain policy prescriptions. Kozma has been involved in some of those negotiations, but the details aren't quite finalized.

Some reproductive rights advocates believe Kozma will use her new position to insert anti-abortion and anti-LGBTQ policies into these agreements.

"[Kozma] is vehemently anti-trans, anti-LGBTQI+, anti-abortion," says Keifer Buckingham, managing director at the Council on Global Equality, a coalition of advocacy organizations that focuses on LGBTQ issues. "For those of us who want to ensure that the provision of U.S. foreign assistance and health doesn't discriminate against people based on who they are, [Kozma's appointment] raises a lot of red flags."

One particular worry is about the Helms Amendment, a U.S. policy that prohibits foreign aid being used to fund abortion services.

"There's been speculation that there's an intention by the U.S. government to expand the Helms Amendment beyond abortion to include LGBTQ's as well," says Musoba Kitui, director of Ipas Africa Alliance, a non-profit that works to provide access to abortion and contraception. He's concerned that health groups that serve those populations could lose funding. That speculation is backed up by reporting from The Daily Signal that the administration is planning to prohibit U.S. aid funding for "gender ideology and diversity, equity and inclusion initiatives."

Given LGBTQ people are often at higher risk for diseases like HIV, such policies could make these communities even more vulnerable, says Kitui.

"We could see more marginalization, inequality, spikes of infection," he says. While many African governments signing these deals understand those dynamics, Kitui says they may still agree to more restrictive conditions as aid cuts have "starved health systems to a point of desperation."

Have information you want to share about ongoing changes at federal health and development agencies? Reach out to Jonathan Lambert via encrypted communications on Signal: @jonlambert.12