The Download of the Week is Large Language Models Are Unreliable Judges by Jonathan H. Choi. Here is the abstract:
Can large language models (LLMs) serve as "AI judges" that provide answers to legal questions? I conduct the first series of empirical experiments to systematically test the reliability of LLMs as legal interpreters. I find that LLM judgments are highly sensitive to prompt phrasing, output processing methods, and model training choices, undermining their credibility and creating opportunities for motivated judges to cherry-pick results. I also find that post-training procedures used to create the most popular models can cause LLM assessments to substantially deviate from empirical predictions of language use, casting doubt on claims that LLMs elucidate ordinary meaning.
Highly recommended.
