2 months ago
19 ARTICLE AD
Within the next several months, Nevada plans to launch a generative AI system powered by Google that will analyze transcripts of unemployment appeals hearings and issue recommendations to human referees about whether or not claimants should receive benefits.
The system will be the first of its kind in the country and represents a significant experiment by state officials and Google in allowing generative AI to influence a high-stakes government decision—one that could put thousands of dollars in unemployed Nevadans’ pockets or take it away.
Nevada officials say the Google system will speed up the appeals process—cutting the time it takes referees to write a determination from several hours to just five minutes, in some cases—helping the state work through a stubborn backlog of cases that have been pending since the height of the COVID-19 pandemic.
The tool will generate recommendations based on hearing transcripts and evidentiary documents, supplying its own analysis of whether a person’s unemployment claim should be approved, denied, or modified. At least one human referee will then review each recommendation, said Christopher Sewell, director of the Nevada Department of Employment, Training, and Rehabilitation (DETR). If the referee agrees with the recommendation, they will sign and issue the decision. If they don’t agree, the referee will revise the document and DETR will investigate the discrepancy.
“There’s no AI [written decisions] that are going out without having human interaction and that human review,” Sewell said. “We can get decisions out quicker so that it actually helps the claimant.”
Judicial scholars, a former U.S. Department of Labor official, and lawyers who represent Nevadans in appeal hearings told Gizmodo they worry the emphasis on speed could undermine any human guardrails Nevada puts in place.
“The time savings they’re looking for only happens if the review is very cursory,” said Morgan Shah, director of community engagement for Nevada Legal Services. “If someone is reviewing something thoroughly and properly, they’re really not saving that much time. At what point are you creating an environment where people are sort of being encouraged to take a shortcut?”
Michele Evermore, a former deputy director for unemployment modernization policy at the Department of Labor, shared similar concerns. “If a robot’s just handed you a recommendation and you just have to check a box and there’s pressure to clear out a backlog, that’s a little bit concerning,” she said.
In response to those fears about automation bias Google spokesperson Ashley Simms said “we work with our customers to identify and address any potential bias, and help them comply with federal and state requirements.”
Privacy and Accuracy
DETR initiated discussions with Google about using AI to process unemployment claims during a sales call a year ago, Sewell said. Over the subsequent months, the agency has run dozens of tests using the company’s technology to analyze hearing transcripts from appeals cases of varying complexity. After determining that Google had created “a solid product and it’s doing the right thing,” Sewell said, DETR agreed to a $1 million contract that was approved by the state’s Board of Examiners last month.
Appeals hearings and the associated documents can contain tax information, social security numbers, and other private identifiers as well as highly sensitive information about a claimant’s health, family, and finances. Under the contract, Google will not have access to personally identifiable information from appeals hearings and will be prohibited from using the confidential data its model processes for other purposes, said Valentina Bonaparte, a spokesperson for DETR.
Bonaparte said Nevada will not be training a new generative AI model for the appeals system. Instead, the state will use Google’s Vertex AI studio, a cloud service that allows developers to fine tune foundation AI models for specific purposes, to create a retrieval-augmented generation (RAG) model. RAG models retrieve information from a specified database—in this case, one containing Nevada unemployment law and previous appeals cases—in order to provide more tailored and accurate results than the foundation model would normally generate.
Carl Stanfield, DETR’s IT administrator, said a governance committee will meet weekly while the model is being fine tuned and then quarterly once it goes live to monitor the system for hallucinations and bias. Generative large language models don’t understand text or reason logically the way a human does, they predict what word or phrase should come next in a string of text based on user prompts and patterns in their training material. Hallucination is an industry term for when those next-text predictions create responses that are factually incorrect or misleading.
In a recent study, researchers from Yale and Stanford universities tested several commercially available RAG models that draw on databases of laws, regulations, and court opinions to help conduct legal research. They found that the models supplied incorrect or misleading answers to questions between 17 and 33 percent of the time and returned incomplete responses between 18 and 63 percent of the time.
Google’s Gemini 1.5 Pro model is currently the best performer on HELM LegalBench, a different benchmarking system that assesses large language models’ ability to answer questions about different aspects of law. Gemini answered legal questions correctly 76 percent of the time in the benchmarking tests, while Gemini 1.5 Flash, a lighter weight version, answered questions correctly 66 percent of the time. Simms said it is too early to say which Google model Nevada will use.
Any lack of accuracy concerns the lawyers with Nevada Legal Services. If the AI appeals system generates a hallucination that influences a referee’s decision, it not only means the decision could be wrong it could also undermine the claimant’s ability to appeal that wrong decision in a civil court case.
“In cases that involve questions of fact, the district court cannot substitute its own judgment for the judgment of the appeal referee,” said Elizabeth Carmona, a senior attorney with Nevada Legal Services, so if a referee makes a decision based on a hallucinated fact, a court may not be able to overturn it.
In a system where a generative AI model issues recommendations that are then reviewed and edited by a human, it could be difficult for state officials or a court to pinpoint where and why an error originated, said Matthew Dahl, a Yale University doctoral student who co-authored the study on accuracy in legal research AI systems. “These models are so complex that it’s not easy to take a snapshot of their decision making at a particular point in time so you can interrogate it later.”
Need for More Speed
Like most states, Nevada’s unemployment system was overwhelmed by an unprecedented number of claims during the pandemic. Following state shutdown orders, businesses sent workers home and closed their doors for months or for good. Congress created the Pandemic Unemployment Assistance (PUA), an entirely new aid program that expanded the number and types of workers eligible for unemployment benefits.
As state agencies struggled to adapt to the influx of claims and new PUA rules, cases piled up and people made mistakes. Claimants filled out forms incorrectly or applied to the wrong unemployment programs, states paid out benefits in the wrong amounts and to workers who weren’t actually eligible. And the longer it took for those mistakes to be resolved in appeals hearings, the more likely it became that unemployed workers couldn’t afford basic necessities or make payments on their homes, their cars, and their credit cards.
In April 2020, Nevada estimated that 30 percent of its workforce was unemployed, the highest rate ever recorded by any state. By 2023, when Sewell took over Nevada’s unemployment agency, there was a backlog of more than 40,000 appeals cases, which has since been worked down to less than 5,000, he said.
Amy Perez, who oversaw unemployment modernization efforts in Colorado and at the U.S. Department of Labor, said that if done correctly, AI automation can address some of the problems that caused life-altering delays for unemployed Nevadans during the pandemic.
The state’s new system is a notable step forward, she said, and one that could be worthwhile if claimants get paid faster, if DETR is vigilant about monitoring the system for hallucinations, and if human referees have the time and support necessary to ensure they can thoroughly review cases.
“There’s a level of risk we have to be willing to accept with humans and with AI,” Perez said. “We should only be putting these tools out into production if we’ve established it’s as good as or better than a human.”
