BenchMIRT: What are LLM benchmarks actually measuring?

BenchMIRT is a method for analyzing what large language model benchmarks actually measure by examining performance at the level of individual questions rather than just overall scores. Benchmarks are designed to test specific abilities like safety or reasoning, but individual questions within them often depend on multiple capabilities at once, and averaging results can hide these mixed signals. BenchMIRT applies a technique from psychometrics called multidimensional Item Response Theory to separate out which underlying capabilities are most closely associated with performance on each question. When applied to 100 language models across 16 benchmarks with over 34,000 questions, the method revealed that some benchmarks measure different capabilities than intended, such as a bias-testing benchmark that aligned more strongly with general reasoning than safety, and it can identify which questions are most informative for evaluation purposes.