Los Angeles County is moving full steam ahead with plans to use its new election equipment for the first time in the upcoming presidential primary.
The system, which includes high-tech "ballot marking devices," has the potential to revolutionize the election industry, creating a transparent and fully accessible way to vote. But for all its innovations, some experts in the voting security community worry it's not ready for prime time.
For starters, the state has yet to sign off on the new technology -- and it's coming down to the wire: In-person voting begins in six weeks, on Feb. 22.
Certification testing has uncovered:
Some candidates for local offices are so disturbed by how ballots appear on the machines that cities like Beverly Hills are exploring lawsuits.
But Dean Logan, the Los Angeles County Registrar Recorder, says his office has worked hard to address and mitigate all concerns.
The issues with the actual voting system come at the same time L.A. County is fundamentally changing not just how but where people vote. Many observers are concerned that shift in voting location alone will lead to widespread confusion.
HOW DID WE GET HERE?
For the past 10 years, Los Angeles County has pursued a unique mission: custom build the state's first publicly-owned and operated voting system and make it secure and accessible for everyone.
To get there, L.A. County election officials used input from thousands of voters and an army of technology advisors, voting advocates, researchers, community stakeholders, and election workers.
Part of the plan has already appeared in voters' mailboxes. Redesigned vote-by-mail ballots for the 2018 midterms introduced a cleaner look, with bubbles to fill out right next to candidates' names instead of the tedious old hunt-for-the-number system.
Now the in-person voting experience is getting an overhaul. The grizzled InkaVote system, first developed in the 1960s, is out. When vote centers open next month, Ballot Marking Devices (BMDs) will greet every voter, letting them make their selections on a touch-screen tablet. Voters will then print out a paper ballot with their choices recorded, review those selections, and cast the vote by feeding the paper right back into the machine.
The full system, from electronic check-in and making vote selections to submitting, processing and tallying ballots, is called "Voting Solutions for All People."
It's Logan's brainchild. The longtime Registrar Recorder for the county said at the outset his goal was to "ensure that... the needs of our voters and the core principles associated with accessible and transparent elections serve as our guide."
Logan and others identified many failures of existing election equipment: much of it in use across the country is outdated, there's little choice in the marketplace, and the systems from private vendors are hard to upgrade, so they're not responsive to the changing needs of voters or local governments.
Most places in the U.S. buy their voting equipment from a small number of companies that are only lightly regulated: A 2016 Wharton School report looked at the business of voting technology:
The industry is dominated by three firms that are moderate in size and neither publicly nor independently held, limiting the amount of information available...about their operations and financial performance.
The guts of these voting machines and their software source code are trade secrets, and difficult for governments to examine. All of this is of growing concern for lawmakers, especially after Russian efforts to probe U.S. election infrastructure during the 2016 presidential race. CEOs of the big three voting vendors testified about election security for the first time before a House subcommittee last week.
Logan's concept has been to create a universal voting experience -- so everyone who goes to a vote center uses the same method of casting a ballot, no matter their physical abilities or language preference. It's a product he's said wasn't available from a private vendor.
"It was designed to provide a comprehensive, secure and accurate in-person voting experience based on significant data collected and user-testing conducted throughout," Logan said.
The new voting system was also billed as an open-source project, with its source code published so anyone can build a copy, tinker with it or use it for voting in another local or state election -- which would be a game-changer in a voting world controlled by private companies.
In many jurisdictions, ballot marking devices are available to accommodate people with vision or hearing problems, while most voters use pen and paper to cast a ballot. Logan believes this creates a "separate but equal type of scenario." Starting with the presidential primary, every in-person L.A. voter must use a ballot marking device, which have audio headphone options, large tactile buttons, and the option to vote in 13 languages.
"Pre-printed ballots will not be available at vote centers," Logan said, adding that voters who want to use pen and paper should request a mail-in ballot by Feb. 25.
The challenge of holding a successful election in L.A. County is colossal: With over 5 million registered voters, this is the largest voting jurisdiction in the country, by a long shot. (If the registered voter population in L.A. was a state, it would be roughly the size of South Carolina.) It is also geographically vast, with diverse cities, neighborhoods and communities spanning nearly 5 thousand square miles. This all makes its elections uniquely difficult to manage.
A decade of work has gone into creating a system to solve that. The county engaged community groups, disability advocates and individual voters in the process of developing its model. The design firm Ideo came up with the look and feel. An elections vendor, Smartmatic, won the contract to build the ballot marking devices and other equipment.
Starting in September, Los Angeles held a series of mock elections to preview the vote center experience and let voters get their hands on the ballot marking devices.
Later in the fall, the machines were used for the first time in real-world voting during L.A.'s Nov. 5 municipal election. Both the mock elections and the November pilot program were "unprecedented" steps to root out glitches and improve the voting experience, according to the Registrar-Recorder's office.
The state also hired independent testing firms to examine the new voting equipment as part of its certification process.
That's when the problems started to pop up, according to reports published by L.A. County and the Secretary of State.
THE PROBLEMS
Here are some of the issues that technical experts and voting rights advocates flagged for LAist after reviewing published independent testing documents. They include possible violations of California Voting System Standards, which govern state certification.
Paper jams that required long reboots to fix bogged down the flow of voting at mock elections and during the November municipal election pilot program, where voters had the choice to use pen and paper or the new ballot marking devices. These problems were described in a Dec. 19th report to the L.A. County Board of Supervisors.
This is a potentially major snag because unlike the relatively small-scale municipal election, the upcoming primary will involve over 30,000 voting machines in use at 1,000 vote centers. Paper misfeeds could mean delays and frustrating lines, and restarting a system in the middle of voting might leave voters questioning the privacy and security of their vote. According to a testing report submitted December 24, the misfeeds were happening at a rate nearly five times what California standards allow.
A state staff report also described voters who "experienced multiple paper jams and misfeeds."
Independent testing firms hired by the Secretary of State turned up critical security flaws in the Voting Solutions for All People equipment, including vulnerabilities that could leave equipment open to hacking or tampering without detection. Two reports, prepared for the state on Dec. 24th, pointed to the presence of USB ports on off-the-shelf PC workstations in the central operating area -- a common feature in the voting industry that could leave the door open to malicious actors introducing malware or booting a system up from an external drive and changing election data.
The testers were able to gain access to electronic event logs to make changes that are impossible to track, and "easily" bypassed seals and locks on all voting system devices. They even got into physical ballot boxes without detection.
And too many people had "root access" on the voting equipment, according to testing reports -- that's a fancy way of saying they had passwords or security privileges to take control of the system and make changes, which ultimately could be used nefariously to alter the voting process.
The elections employees are also new to this equipment -- and they may not be ready. Independent reports, the mock election and L.A.'s November pilot program exposed shortcomings with the training and capabilities of voting center staff to properly guide voters through the brand new, high-tech process of voting using the ballot marking devices.
On a macro level, despite promising an open-source system, experts in the voting technology space say the county hasn't popped the hood for the broader election security community to take a look at the underlying code or engineering information.
"System developers have not shared engineering documents with the public and have rebuffed any attempts to learn technical details about the system architecture and underlying technology," said Richard DeMillo, Director of the Center for Information Security Research at Georgia Tech.
It's a sentiment Logan's collaborators don't share. Whitney Quesenbery, cofounder of the Center for Civic Design, took time at a public hearing held by the Secretary of State on Friday to praise the "degree of openness, the inclusiveness of the research" the Registrar Recorder's office conducted. Quesenbery is on a technical advisory committee for the project, and described "many hours we spent debating almost every technical issue that I can think of in the user interface."
But DeMillo called the state testing results "alarming."
"Had there been an open review of engineering documents earlier in the process, defects like these would have been recognized and addressed," DeMillo said.
WHAT THE COUNTY SAYS
Logan says the certification process involves a lot of communication -- and negotiation -- between state and county election officials.
He told LAist in an email that his office went the extra mile to troubleshoot ahead of the March 3 election. And since the state tests, "changes were made to the system and have been through further review."
Logan has also made clear that his office has identified and mitigated any realistic risks to election integrity or voting security.
"This is arguably the voting system that has had the most robust testing and scrutiny of any that's been presented before," Logan said at the Jan. 9 hearing on state certification. "I...say that as a statement of fact and a statement of pride."
The certification is on schedule, according to the Registrar's office, and the issues brought up in mock elections, the November pilot program and certification testing have not caused delays. The county conducted several rounds of its own hardware and software testing ahead of the Secretary of State's investigation.
The state testing reports "were reviewed in detail with the County and our system integrator/manufacturer," Logan said in an emailed response. For some problems, "revisions and refinements to the system were made," he said.
For other issues, Logan added, the county provided documentation "to clarify some issues and information detailing procedural, operational and environmental use conditions." In other words, there's a dialogue going on between the county, state, and testing firms about what actually happens in vote centers, to make sure the tests -- and the risks they identified -- accurately reflected real-world scenarios.
For example, during actual voting, election workers will be present at all times to assist voters and to prevent unauthorized access to election equipment.
"An ongoing review and exchange of that information is standard in the testing and certification process," Logan said.
The county has also made hardware changes to mitigate paper jams, added plugs to exposed USB ports, and refined procedures involving tamper evident seals on ballot boxes, Logan said.
He stressed the final certification decision will be made by the state.
In a staff report, the Secretary of State's Office of Voting Systems Technology Assessment endorsed the new system, saying it "meets all applicable California and federal laws."
"The County and our manufacturer are committed to addressing any outstanding questions from the State and complying with any required conditions identified through the certification process," Logan said.
Any changes made after the November 5 pilot program would be last-minute, according to some experts -- suggesting L.A. County could be rushing things over the finish line.
"That doesn't leave a lot of time for testing and validation of the changes, particularly when it's bumping up against a January certification deadline," said Eddie Perez, Global Director of Technology Development for the OSET Institute. Perez formerly worked for one of the big three election vendors, Hart InterCivic, as Director of Product Management and Certification.
"It would be a high-risk race to certification, and to implement on this scale in a Presidential year for any county -- and for a county of 5.4 million voters, with a brand new voting system, and new vote centers, the risks are greatly magnified."
"It's uncommon to have this much uncertainty this late in the game," Perez added.
What about poorly trained election workers getting caught off guard when actual voting starts?
Employees will be "comfortable and knowledgeable" to help voters thanks to multiple online and in-person training sessions, said Registrar-Recorder spokesman Michael Sanchez in an email.
"In addition to the formal training received prior to the election, the 11-day voting period will also serve as real-time hands-on training for our Election Workers," Sanchez said. "[W]hen voting ramps up to Election Day, our Election Workers will have several days of experience and will be able to provide a positive voting experience."
Experts agree: no voting system is perfectly un-hackable.
"Election officials are risk managers," said Marian Schneider, president of the nonprofit group Verified Voting. "Things happen. Elections are messy. But they have to try to get that risk as close to zero as possible."
MACHINES V.S. PEN AND PAPER
Dean Logan says offering paper ballots everywhere doesn't achieve the goal of providing a smooth voting experience to everyone in L.A. County.
"It is not practical to have copies of all ballot styles in 13 languages at roughly 1,000 voting locations where Ballot Marking Devices that produce voter-marked, human-readable paper ballots are available to all voters," he said in an email.
But some security experts question any machine that gets between a voter and their ballot.
The issue pits the goals of disability rights advocates and election officials like Logan, who champion an accessible voting experience, against security hawks who fear the possibility of election interference going on under our noses, without detection.
The county's new machines are air-gapped -- meaning they're not connected to the internet or any network -- and no election data is stored on the device. Voters print out their choices at the end of the process and cast the paper ballot, and it's that physical record that's actually counted.
But all ballot marking machines have the potential to be tampered with, no matter how much security is built into the hardware and software, or how often people are encouraged to carefully check their printed paper ballots. Research shows the vast majority of voters don't notice problems with ballots created on machines that have been "hacked."
"Not offering voters the opportunity to hand mark a ballot in polling places or vote centers is a serious blow to election integrity," said Philip Stark, professor of statistics at the University of California, Berkeley. Stark led the first statewide risk-limiting election audit in the U.S., a practice considered the "gold standard" for election security.
He's among the security experts who believe ballot marking devices should only be used for situations where disabilities or language accessibility requires them. There's even a bill floating around congress to mandate hand-marked paper ballots in most situations.
Orange County, for example, is giving in-person voters the option to use ballot marking devices if they need to, or just prefer it - but everyone else is encouraged to use paper and pen. Orange County's equipment has also been used by Hawaii, Virginia, Texas and hundreds of counties, so this is not the first time it'll be implemented with real voters.
WHAT DOES IT MEAN THAT THE SYSTEM MUST BE 'CERTIFIED'?
New voting machines have to pass a gauntlet of tests mandated by California's Secretary of State before they're certified for widespread election use. These check for compliance with the California Voting System Standards.
The Secretary of State's certification process puts a system's hardware, software, usability and security components under a microscope.
The tests -- usually run by independent firms -- simulate the risks and large volume that come with a live election. Testers also try their best to hack the election hardware and software or tamper with ballots.
The bar is supposed to be higher in California. The state's certification requirements are considered to be stricter than the federal standards.
The federal government doesn't mandate paper ballots, for instance, while California requires a voter-verifiable paper trail that could take the form of a ballot or a receipt. L.A. County will retain paper ballots after they are counted for auditing.
Election security experts say it's simple: paper ballots can be audited, and a paper trail is necessary to ensure the integrity of the vote. L.A. County's new system relies on a paper ballot that voters should double-check before feeding it back into the machine to cast their vote.
The list of voting systems certified for us in California elections is here. The most recent is the Verity Voting system from Hart Intercivic -- the voting machine vendor for Orange County and hundreds of other U.S. jurisdictions. The state approved that technology on Dec. 27.
LOCAL PUSHBACK
Worries are mounting as more election officials and campaigns around L.A. County get a preview of the ballot marking devices voters will be using in the March primary.
Candidates in city council, Assembly, and State Senate races -- the contests that don't rotate ballot order -- are worried voters could miss their names entirely.
The digital touch-screens display just four candidates at a time, and the button that moves a voter down the ballot is too easy to overlook, critics say. Some critics worry that voters could mistakenly skip past any candidates listed lower on the ballot.
The Beverly Hills City Council even voted last week to explore a lawsuit against Los Angeles County and the Secretary of State, asking them to fix the feature.
Los Angeles County made changes to the "MORE" button to make it more visible after the fall's mock election.
In a letter responding to the Beverly Hills City Clerk, Logan defended the design -- he said it worked properly in L.A.'s November municipal election:
"Prior pilot election results and findings further indicate that users were able to recognize and use the "MORE" button to navigate pass the initial screen to view the names of all candidates on the BMD, and that the candidates receiving the fewest votes were consistent across all three ballot types (BMD, vote by mail, and Inkavote).
"Taken altogether, we believe there is strong evidence to show that voters can recognize and use the "MORE" button as currently designed to scroll to the remaining candidates before making his or her desired election in a contest. There are also other factors and timing issues limiting our ability to modify the VSAP system, as further addressed below."
[Read more here on LAist: Beverly Hills Might Sue Over LA County's New Voting Machine Design]
SO. MANY. CHANGES.
As if the issues with the voting machines are not enough, running an election in Los Angeles is no piece of cake, even in a normal year. But this massive technical overhaul is happening during the same election that brings a huge shift in where and when Angelenos vote as the county adopts new voting locations to comply with the Voter's Choice Act of 2016.
What it means: Neighborhood polling places like your local high school gym or church community room are history. Election "Day" will really be an 11-day voting period leading up to March 3, when in-person voters will head to new "vote centers." These centers will replace the traditional polling place. They'll be open for longer hours, allowing people not only to cast ballots but to register to vote, change parties, replace their ballots, or get help in over a dozen languages.
Five California counties that switched to vote centers in 2018 saw a slight voter turnout boost. But there are plenty of pitfalls waiting in L.A. County, where voters are much more likely to vote in-person compared to the rest of the state.
The worst-case scenario outcome is that despite messaging campaigns about vote centers, people will show up to their old polling places on election day - and no one will be there.
The county is investing in advertising to get the word out. But the physical locations of "vote centers" in L.A. County where people will be voting starting Feb. 22 have not been announced yet.
County election officials say ultimately, all the voting transitions will lead to a better 2020 election.
Michael Sanchez, a spokesman with the Registrar-Recorder's office, said in an email: "Our office has complete confidence the [new voting] model will significantly improve the voting experience for all voters."
