Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation
Second author Lead authorPublished
ICLR 2026, main conference · 2026
- In brief
- Replaces the Pass@k score used to compare language models with a Bayesian estimate that comes with error bars.
- Details
- Outcomes are modeled as categorical with a Dirichlet prior, giving a closed-form posterior mean and uncertainty for any weighted rubric. Compared with Pass@k and avg@N, the estimate converges faster and gives more stable rankings at far smaller sample counts.
- Why it matters
- Gives benchmark rankings an explicit finite-sample uncertainty, so a difference between two models can be judged against its credible interval.
- My contribution
- As a lead author, co-developed the Bayesian framework for LLM evaluation, including posterior estimates and credible intervals for finite-sample uncertainty.
BibTeX
@inproceedings{Hariri:2026passk,
author = {Hariri, Mohsen and Samandar, Amirhossein and Hinczewski, Michael and Chaudhary, Vipin},
title = {Don't {Pass@k}: A {Bayesian} Framework for Large Language Model Evaluation},
booktitle = {Proceedings of the Fourteenth International Conference on Learning Representations (ICLR)},
year = {2026},
eprint = {2510.04265},
archivePrefix = {arXiv},
primaryClass = {cs.AI}
}