California reveals use of six high-risk AI systems, up from zero last year

California’s state government went from claiming it used zero high-risk AI systems to disclosing six of them in the span of a single year. Not because it suddenly adopted six new tools. Because it finally started counting honestly.

The Department of Technology published its updated inventory on June 15, revealing half a dozen automated decision systems that directly affect people’s lives in areas like criminal justice and public benefits. Several of these systems have been operational for years, quietly shaping outcomes for incarcerated individuals and unemployment insurance applicants while officially not existing on any transparency ledger.

What California is actually using

The six newly disclosed high-risk systems include tools designed to predict recidivism among incarcerated people and applications that evaluate potential unemployment insurance fraud.

Adding another layer to the story, six additional systems were initially flagged as high-risk during this year’s review but were subsequently reclassified to a lower risk tier. So the state identified 12 potentially problematic systems, then decided half of them weren’t actually that concerning.

The law that forced the disclosure

None of this transparency happened voluntarily. It’s the product of AB 302, a California law enacted in October 2023 that requires state agencies to conduct annual inventories of their automated decision systems and report on high-risk uses. The law mandates that agencies report annually to legislative committees through 2029, providing details about the systems’ functions, the data they use, benefits they provide, and strategies for risk mitigation.

The first inventory, published roughly a year ago, reported zero high-risk systems across all state agencies. The jump from zero to six doesn’t necessarily mean California suddenly deployed a bunch of new AI tools. The more likely explanation is that agencies improved their internal evaluation processes, or that the Department of Technology applied stricter classification standards the second time around.

AB 302 doesn’t ban high-risk AI use outright. It requires disclosure so that legislators, journalists, and the public can scrutinize what’s happening.

What this means for the AI governance landscape

The recidivism prediction and fraud detection applications disclosed by California sit at the intersection of AI’s greatest promise and its most documented failures. The fact that California is now at least acknowledging their existence is progress. Whether the state will go further and publish performance metrics, audit results, or demographic impact analyses remains the more important question, one that AB 302’s reporting requirements may not yet be equipped to answer.

Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our Editorial Policy.