LAUSD’s Stunning Reading Score: Illusion, Real Or Something In Between?
Los Angeles Unified’s unexpectedly big increase in eighth grade reading on the National Assessment of Educational Progress drew praise and skepticism after the release of the scores last month. The results may have warranted some of both.
LAUSD was the only one of the 26 urban districts nationwide that took NAEP to see improvement in 2022. And its nine-point growth from pre-pandemic 2019 was by far the biggest increase among all states and school districts on any test. Most saw a significant decline in reading and record declines in math on the fourth and eighth grade tests.
Some data experts and testing analysts questioned the size of L.A.’s gain and the significance of any single year’s score in isolation. In 2019, the district had an unusually large drop in scores from 2017, putting the rebound in 2022 into context, said Brian Gill, a senior fellow and data expert at Mathematica in Princeton, New Jersey. There may be improvement in reading in the district, or there may be an unexplainable quirkiness in the data; that can happen with statistical outliers, he said. “A focus on outliers could cause you to get fooled by randomness.”
But Los Angeles Unified Superintendent Alberto Carvalho took offense at these criticisms. Carvalho, the former superintendent of Miami-Dade County Public Schools for the 14 previous years, had just started to work in L.A. while NAEP was administering the tests. The testing period was between January and March, and Carvalho's first day was Feb. 14.
During a webcast in Washington, D.C. the day after the scores’ release, Carvalho attributed many factors to L.A.s’ success, among them higher levels of middle school attendance and engagement, rapid universal distribution of home computers during the pandemic, high-intensity tutoring and a “massive amount” of teacher training.
In a Nov. 10 tweet, he lashed out at doubters, likening them to flat-earthers: “When the validity of released ‘gold standard’ #NAEP data, after the detailed due diligence conducted to ensure it, is questioned, you might as well question sea-level rise, global warming, or gravity for that matter; then again some of these voices probably do just that.”
States and urban districts put a lot of faith in NAEP because it is the one common national test they take. Usually it’s given every other odd year, but because of the pandemic the 2021 test was pushed out to 2022.
State tests preclude direct comparisons because states create their own tests with different methodologies and measures of progress. An exception is the Smarter Balanced assessment, which California and 10 other states use.
Unlike Smarter Balanced, where gains and losses are commonly measured in percentages, NAEP changes over time are measured in points on a 500-point scale; for states, differences of three points are often deemed significant.
Los Angeles’ nine-point increase in the eighth grade reading test stood out. It happened in a year that NAEP scores in reading and especially math fell nationwide. The average score in eighth grade reading among the 26 urban districts dropped three points.
Unlike state assessments, which the federal government requires all students in grades three to eight to take — more than 3 million in California — NAEP is given to far fewer students: hundreds in small states and about 4,000 in California. NAEP selects schools and students to reflect the demographics of states and districts.
About 1,400 fourth and eighth graders in L.A. took the test this year. Up to two dozen students from each grade were chosen from among sets of 60 schools. Schools were alerted in the fall that they would give the test, but they didn’t get to choose the students.
To limit the testing time required, students took only about 20% of the tests. For eighth grade reading, each student was assigned two of the 12 sections, each a half-hour long.
LAUSD’s eighth grade reading scores grabbed the attention of the National Center for Education Statistics, the federal agency that administers the test. Its post-mortem found the methodology was sound and there wasn’t a high rate of absences, said Grady Wilburn, a statistician with the national center. The center also asked the American Institutes for Research to replicate the results. In its unpublished report, the institute found no flaws in the data and that high enrollment loss during the pandemic — another variable — didn’t affect the scores, Wilburn said.
However, the analysis did find one significant disparity. The 2019 Los Angeles Unified NAEP assessments didn’t include the scores of affiliated charter schools, while other testing years, including the 2022 test, did. Unlike Los Angeles’ 300 independent charter schools, the 50 affiliated public schools are run by the district, and teachers fall under the union contract.
The district’s seven affiliated charter middle schools, mostly located in prosperous neighborhoods, are high achievers. On this year’s eighth grade Smarter Balanced English language arts tests, 66% of students on average made or exceeded the standard, compared with the district average of 41%.
In 2022, NAEP tested students in five of the seven schools. They apparently did well on NAEP, too. The national center attributed two of the district’s nine-point increase to the affiliated charters alone.
That still leaves a seven-point gain for the district, compared with no change in California’s score from 2019. There was a similar pattern, to a lesser degree, in the 2022 Smarter Balanced English scores for eighth grade language arts: Scores for the state declined nearly 3 percentage points from 2019, while Los Angeles’ gained about 1 percentage point.
What distinguished LA?
Austin Beutner, who was superintendent of LAUSD from May 2018 through June 2021, is confident that actions the district took during the pandemic led to improvement in the NAEP scores, although he acknowledges establishing cause and effect is problematic. “In public education, people want to attribute cause where it doesn’t exist, and they miss it where it does,” he said.
What is clear, he said, is that L.A. Unified moved faster than most urban districts to create conditions conducive to learning despite the ravages of COVID-19 and unemployment in low-income areas. All students got computer and internet service, and the district set up the nation’s biggest and most cost-effective coronavirus testing system, followed by vaccinations, he said. It created the largest school food program for families. “COVID testing kept people safe and helped get teachers and students back into school classrooms and keep them there,” he said.
Middle school is where you would see the difference that a free computer and internet made, he said. “For the first time, their learning could go from school to home and back and forth in a way it never had before.”
There were intangible factors that made learning fun and engaging. The district worked with PBS to create three TV channels, by grade, during the year of remote learning, watched by a quarter of a million people weekly, Beutner said. The musical instrument maker Fender distributed 15,000 guitars for free for after-school lessons for middle schoolers, who could also learn about animations and storytelling through a partnership with Illumination Entertainment.
But the biggest impact, Beutner said, was in literacy and numeracy skills for kindergarten to third graders through Primary Promise, an initiative that placed 300 reading specialists and 140 math specialists in high-need elementary school classrooms. They provided small-group instruction, at first on Zoom and then in person, expanding to hundreds of schools and thousands of students. The first group of 2,500 first graders was chosen because they were struggling academically, with only 9% reading at grade level, Beutner said. Within 12 weeks, 42% of these same students were at grade level, he said.
“To me, literacy is the holy grail of public education, and Primary Promise is the biggest story of what we did,” he said.
These strategies may hold promise, but they didn’t make a huge difference on the fourth grade NAEP reading score. It was flat, while the nation, state and the urban districts’ average dropped two to three points. On the Smarter Balanced English language arts test, the district’s and the state’s scores both fell 4 percentage points.
Alison Yoshimoto-Towery, who was L.A. Unified's chief academic officer from 2019 until this September, said important changes that were less visible to the public included the adoption in 2019 of a second literacy curriculum for K-6 — Core Knowledge Language Arts or CKLA — which is praised by advocates of “science of reading” for its attention to phonics and other foundational skills.
That adoption didn’t factor into the eighth grade NAEP reading scores, but the introduction in 2021 of diagnostic or formative assessments and digital literacy tools could have had an impact, she said. Formative assessments provide status reports on students’ strengths and weaknesses, to guide teachers’ instruction, which the summative Smarter Balanced tests cannot do.
“Many things together resulted in a stronger foundation for teaching and learning,” she said, including increasing access to virtual and in-person field trips that spark student interest.
However, other claims of good attendance and widespread high-dosage tutoring appear to be a stretch. Intensive after-school tutoring in 2021-22 was largely a failure; money budgeted for it in schools went unused. And more than 30% of students in the district were chronically absent last year.
Likelihood of statistical errors
Morgan Polikoff, a professor at the USC’s Rossier School of Education, said he was withholding judgment on what might have led to the district’s gain on NAEP. “I don’t want to hold up anyone as a model for post-pandemic response right now," he said. "The jury is still out."
The progress in eighth grade test scores could instead reflect the impact of the Local Control Funding Formula, a law California adopted in 2013. It shifted funding to districts based on enrollment of English learners, low-income students and homeless and foster children. “Research shows funding matters, and Los Angeles gained the most,” he said.
Polikoff, Gill of Mathematica and other researchers trust NAEP’s accuracy and its methodology.
“I find no problem with the number of kids tested for NAEP,” said Gill. “I think they are testing enough to produce useful results if interpreted appropriately.”
Problems arise when comparing states and districts with different sample sizes and levels of precision, Gill said. NAEP statisticians may determine that a three-point drop in states’ reading scores is significant, while a three-point drop in urban districts’ scores is not. But that does not mean that urban districts outperformed the nation, Gill said.
NAEP procedures for a representative sample of students are designed to eliminate bias, but they cannot completely eliminate random errors caused by testing conditions, unpredictable circumstances and differences in individual student test-takers from year to year.
In 2019, LAUSD’s NAEP scores fell a rare six points. NAEP test-taking overlapped with a six-day teachers' strike and the weeks long tension that led up to it. The possible impact offers a different perspective on the seven-point rebound in 2022.
“The strike took time and attention away from the classroom. I’m surprised nobody thought of it when they looked at NAEP scores that year,” Beutner said.
For two decades, as national reading scores on NAEP stagnated, then declined, L.A. Unified showed slow, steady progress on eighth grade reading with the exception of one year, 2019. This year’s score, three points above 2017, was consistent with the pattern. For the first time, its score was higher than the urban school average. In 2003, it lagged the state by 17 points and the nation by 27 points. This year, the gap had closed to two points. Modest improvement may be the good news.
“The most likely reality is that whatever gain actually happened in L.A. was more modest than nine points, and that it probably had something to do with changes in the district over time — instructional or other policy changes,” said Polikoff.
Beutner has a different take. “On a relative basis, apple to apple, NAEP to NAEP, L.A. did meaningfully better than other major school districts” in 2022, he said. “But nationwide the NAEP numbers haven’t changed in 30 years. That’s the bigger story.”