BenchMIRT
HoldTechniques
A method for auditing LLM benchmarks at the individual prompt level using multidimensional item response theory.
Why it's here
Placed in Hold: 1 article(s) of evidence from 1 source(s), led by research-stage coverage, with 0 in the last 30 days. Confidence 24%. Low accumulated evidence, so it defaults conservatively pending more signal.
Evidence (1)
- 7Hugging Face Blog·9/1/2026researchBenchMIRT audits what LLM benchmarks measure
Hugging Face highlights BenchMIRT, a new auditing method that uses multidimensional item response theory to analyze LLM benchmark prompts at the item level. Applied to results from 100 models across 16 benchmarks, it found two stable underlying dimensions—safety and general reasoning—and showed that some benchmarks, such as BBQ, may measure different capabilities than their stated purpose suggests.