Why figuring out how many homes California needs is more art than science

Imagine you’ve finally taken your car to the mechanic to investigate that mysterious warning light that’s been flashing on your dashboard for the past week and a half.

The mechanic informs you that your car's brake fluid is too low. Dangerously low. Your brake fluid supply, he says, has reached “crisis” levels, which sounds both scary and very expensive.

Naturally, you would prefer that your car have a non-critical amount of brake fluid. “How much more do I need?” you ask.

“A quart,” the mechanic responds. “No, actually, three quarts. Or maybe seven gallons — but only routed to your rear brakes. Actually, let’s settle on half an ounce.”

Such is the situation with California’s housing shortage.

For nearly a decade now, the Legislature has been churning out bills, Attorney General Rob Bonta has been filing lawsuits and Gov. Gavin Newsom has been revamping agencies, dashing off executive orders and quoting Ezra Klein with the explicit goal of easing the state’s chronic undersupply of places to live.

California simply doesn’t have enough housing and this shortage is the leading cause of our housing affordability concerns — virtually everyone in and around the state government, along with the vast majority of academics who have studied the issue, seems now to agree on this point.

This consensus was on display this year when lawmakers passed two sweeping changes to state housing law, one that shields apartment developments from environmental litigation and the other that would permit denser development near major public transit stops in big cities. Both were legislative non-starters just a few years ago. These days even the opponents of these bills have accepted the premise that the state faces a “housing shortage,” a term evoked at least 30 times in committee hearings and floor speeches this year.

Now, if only anyone could agree on how big the housing shortage actually is.

Plenty of people have tried to put a number on the problem.

In 2015, the Legislative Analyst’s Office, which serves as a policy analysis shop and think tank for the Legislature, took an early crack at quantifying the state’s shortage by calculating how many additional units major metro areas would have had to build over the prior three decades to keep housing cost inflation on par with that of the rest of the country.

It came up with 2.7 million missing units.

A year later, consulting giant McKinsey one-upped the LAO, putting the state’s “housing shortfall” at 3.5 million houses, apartments and condos, a number Newsom campaigned on.

Not all estimates hit seven digits. In 2024, the housing policy nonprofit Up For Growth published the more modest estimated shortfall of 840,000 units, which comes pretty close to the 820,000 Freddie Mac put forward a few years earlier.

California Housing Partnership, a nonprofit that advocates for affordable housing, has counted the deficit at 1.3 million units — but not just any units. That’s how many homes the state needs to add that are affordable to people making under a certain income.

Then, this summer, a group of housing analysts including an economist at Moody’s Analytics, came up with the strikingly low figure of just 56,000 — though the authors acknowledged that it’s probably an underestimate.

Estimates of the nation’s overall housing supply are similarly all over the place: From as high as 8.2 million to 1.5 million (and, in one controversial paper, zero).

What even is a housing shortage?

The concept of a “housing shortage” is, in theory, pretty simple, said Anjali Kolachalam, an analyst at Up For Growth.

“It’s basically just the gap between the housing you have and the housing you need,” she said.

In practice, defining and then setting out to quantify the “housing you need” is an exercise fraught with messy data, guestimation and an inconvenient need for judgement calls.

Most estimates begin with a target vacancy rate. In any reasonably well-functioning housing market, the logic goes, some houses and apartments sit empty, either because they’re between renters, they’ve just been built or sold, they’re being fixed or renovated or they’re someone’s second home. A modest vacancy rate is what allows you to pull up Zillow or Craigslist and not get a “No Results Found” error. A very low one suggests there aren’t enough homes to go around.

But choosing a “healthy” vacancy rate — one that reflects a functional housing market — and then backing out the number of additional homes needed to hit it, is more art than science. Most estimates turn to historical data to find some level when supply and demand weren’t completely out of whack. Whether that halcyon period of relative affordability is 2015 or 2006 or 2000 or 1980 varies by researcher and, likely, by the region being considered.

Beyond that, many researchers have tried to put a value on what is sometimes called “pent up” demand or “missing households.” Those are all the people who would have gone off and gotten their own apartment or bought their own place, but, because of the unavailability of affordable places to live, have opted to keep living with housemates, with parents or, in more extreme cases, without shelter of any kind.

Absent a survey of every living person, there’s no way to precisely measure how many people fall into this camp.

“This notion of ‘pent up demand’ is necessarily in an economist’s judgment call,” said Elena Patel, a fellow at the Brookings Institution who helped put together a nationwide shortage estimate last year (4.9 million).

These variations in methods help explain some of the differences in the shortage estimates. Other differences pop-up thanks to the vagaries of data.

The Moody’s Analytics-led report, for example, calculated a national shortage of roughly 2 million units by adding together both the number of new units needed to raise the overall vacancy rate and the homes needed to backfill their measure of “pent up” demand. But for its California-specific estimate, the data wasn’t available to do the latter, potentially leaving out a big chunk of the statewide shortage.

Then some estimates differ because the analysts are defining the shortage in a completely different way.

The California Housing Partnership looks at the difference between the number of households deemed by federal housing guidelines to have “very” or “extremely” low incomes and the number of units that those households could conceivably rent with less than 30% of their incomes.

That gap of 1.3 million gets at a problem totally distinct from an overall shortage of homes.

Finally, there’s the question of scale. Housing markets are, on the whole, local. A national shortage is going to add together San Francisco and Detroit, masking the extremes of both. A shortage estimate for a state as large and diverse as California may have the same problem.

“It is like looking for a weather forecast for a trip to the beach and being told that the average temperature nationwide is likely to be 67 degrees,” the authors of the Moody’s-led analysis wrote.

Why estimate a shortage?

What might be more valuable than fixating on any one shortage estimate, said Daniel McCue, a researcher at the Harvard Joint Center for Housing Studies, is to look at all the estimates together and appreciate that, by and large, they’re all huge.

“Whether it’s one-and-a-half million or five-and-a-half million, these are big numbers,” he said. That leads to an inescapable takeaway, he said. “There’s so much to do. There's so far to go.”

Patel, from Brookings, said trying to put a precise tally on what is ultimately the somewhat nebulous concept of a “housing shortage” is still a worthwhile exercise because it gives lawmakers and planners a benchmark against which to measure progress.

How much additional taxpayer money should a state throw at affordable housing development? How aggressive should a locality be in pursuing changes to local zoning? “The more concrete you can be in policy making land, the better,” she said.

The State of California does in fact have its own set of concrete numbers.

Every eight years, the Department of Housing and Community Development issues planning goals to regions across the state — a number of additional homes, broken down by affordability level, that every municipality should plan for. These are, effectively, California government’s official estimates of the state shortage.

To cobble together these numbers, state regulators look at projections of population growth to accommodate the need for future homes and then tack on adjustments to account for all the homes that weren’t built in prior periods, but perhaps ought to have been. If a region has an excess number of households deemed overcrowded, it gets more units. If vacancy rates are below a predetermined level, it gets more units. If there is a bevy of people spending more than 30% of their incomes on rent, more (affordable) units.

It’s a process that the state regulators have come to take somewhat more seriously in recent years, engendering an ongoing political backlash from density-averse local governments and neighborhood activists.

In the state’s last estimate, the topline total was 2.5 million units.

This coming cycle, which has already begun in the rural north and will slowly roll out across the state in the coming years, will produce yet another number. That will be one more estimate for state lawmakers of how much brake fluid the car needs.

This article was originally published on CalMatters and was republished under the Creative Commons Attribution-NonCommercial-NoDerivatives license.