Publications

Don't Pass@k: A Bayesian Framework for Large Language Model Evaluation

Second author Lead authorPublished

M. Hariri, A. Samandar, M. Hinczewski, V. Chaudhary

ICLR 2026, main conference · 2026

In brief
Replaces the Pass@k score used to compare language models with a Bayesian estimate that comes with error bars.
Details
Outcomes are modeled as categorical with a Dirichlet prior, giving a closed-form posterior mean and uncertainty for any weighted rubric. Compared with Pass@k and avg@N, the estimate converges faster and gives more stable rankings at far smaller sample counts.
Why it matters
Gives benchmark rankings an explicit finite-sample uncertainty, so a difference between two models can be judged against its credible interval.
My contribution
As a lead author, co-developed the Bayesian framework for LLM evaluation, including posterior estimates and credible intervals for finite-sample uncertainty.
BibTeX
@inproceedings{Hariri:2026passk,
  author        = {Hariri, Mohsen and Samandar, Amirhossein and Hinczewski, Michael and Chaudhary, Vipin},
  title         = {Don't {Pass@k}: A {Bayesian} Framework for Large Language Model Evaluation},
  booktitle     = {Proceedings of the Fourteenth International Conference on Learning Representations (ICLR)},
  year          = {2026},
  eprint        = {2510.04265},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI}
}

All publications