Judges around the world have made headlines for illicitly using generative AI in their work. But in Pakistan, a large-scale trial of a specially designed AI tool for judges found the technology—together with appropriate training–boosted the number of cases resolved by 6.3 percent with no obvious drop in the quality of judgments.

With a backlog of 2.26 million cases and fewer than two judges per 100,000 people—compared to 22 in the EU and eight in Brazil—Pakistan’s judiciary was in sore need of help. So, in consultation with the judiciary, economist Sultan Mehmood, of the New Economic School in Moscow, and collaborators tested whether AI could ease the burden.

They built a custom tool combining OpenAI’s GPT-4 large language model (LLM) with a knowledge base of nearly 130,000 Pakistani judicial opinions and statutes, to help judges with legal research and drafting judgments. They began offering the tool in 2024 to 1,559 trial judges—roughly half the country’s justices.

“We do find an increase in cases resolved, and we don’t find any corresponding decrease in decision quality,” Mehmood says.

“It’s pretty amazing that he’s able to pull this off,” says David Autor, an economics professor at MIT. “It’s not easy to do large-scale field experiments in civil service, but especially where the stakes are so high.” The 6.3 percent productivity boost is not overwhelming, he says, but it’s credible and likely to improve as the tool is more widely used.

S. law professor Eric Posner has compared LLM judgments to human judgments in a single case study .

Source link