# AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment

- Authors: Dario Loi, Elena Maria Muià, Federico Siciliano, Giovanni Trappolini, Vincenzo Crisà, Peter Kruger, Fabrizio Silvestri
- Venue: Preprint, October 2025
- Published: October 26, 2025
- Page: https://dario-loi.github.io/publications/autobench/
- Markdown: https://dario-loi.github.io/publications/autobench/index.md
- PDF: https://dario-loi.github.io/publications/autobench/autobench.pdf
- arXiv: https://arxiv.org/abs/2510.22593
- Related: AutoBench (https://autobench.org/)
- Topics: Benchmarks (https://dario-loi.github.io/publications/topics/benchmarks/)
- Keywords: LLM evaluation, Large language models, LLM-as-a-judge, Peer assessment, Automated LLM benchmark, Benchmark contamination, Benchmark saturation, LLM leaderboard, Multi-judge evaluation, MMLU-Pro, GPQA
- How to cite: Dario Loi, Elena Maria Muià, Federico Siciliano, Giovanni Trappolini, Vincenzo Crisà, Peter Kruger, and Fabrizio Silvestri (2025). AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment. arXiv:2510.22593. https://arxiv.org/abs/2510.22593

Summary: Language models take turns writing tasks, answering them, and judging each other's answers. Weighting each judge by how reliable it proves yields rankings that agree with established benchmarks.

Abstract: We present AutoBench, a fully automated and self-sustaining framework for evaluating Large Language Models (LLMs) through reciprocal peer assessment. This paper provides a rigorous scientific validation of the AutoBench methodology, originally developed as an open-source project by eZecute S.R.L. Unlike static benchmarks that suffer from test-set contamination and limited adaptability, AutoBench dynamically generates novel evaluation tasks while models alternately serve as question generators, contestants, and judges across diverse domains. An iterative weighting mechanism amplifies the influence of consistently reliable evaluators, aggregating peer judgments into consensus-based rankings that reflect collective model agreement. Our experiments demonstrate strong correlations with established benchmarks including MMLU-Pro and GPQA (respectively 78% and 63%), validating this peer-driven evaluation paradigm. The multi-judge design significantly outperforms single-judge baselines, confirming that distributed evaluation produces more robust and human-consistent assessments. AutoBench offers a scalable, contamination-resistant alternative to static benchmarks for the continuous evaluation of evolving language models.

Key figure (Figure 1): Diagram of the AutoBench framework. The AutoBench loop in three stages. One: a model generates a task and the other models validate it. Two: every model answers. Three: the models rank one another's answers into a judgment matrix, which, weighted by each judge's reliability, updates the model scores and the judge weights for the next round. An inset shows a contestant and a judge joined by one judgment.

Every few months a new language model tops a leaderboard. Every few months someone asks whether the number still means anything. Part of the doubt is practical. A benchmark that is public for long enough can leak into training data, and a test that every strong model passes no longer separates the strong from the stronger. Part of it is structural, because each fix we know has moved the dependency somewhere else: toward human annotators who write the tasks, toward a fixed panel of judge models, or toward one judge, e.g., GPT-4, that serves as the default reference.

AutoBench asks what happens if no outside party is needed. The models being ranked write the tasks, answer them, and grade each other. The grades of models that prove reliable count for more. This is a form of peer review between language models. We wanted to know whether a ranking obtained that way agrees with established benchmarks.

## Why LLM-as-a-judge needs more than one judge

Grading one language model with another is now routine and cheap. A single judge adds its own bias to every score. A model that grades answers may also rate its own higher. The judgment matrix in the paper makes this inspectable, since its diagonal records how each model rates its own answers, so self-preference is a visible quantity. Spreading the grading over every participant dilutes the bias of any one judge.

In practice, this is multi-judge LLM evaluation with a feedback loop, i.e., a model's weight as a grader depends on how reliable it proves. The structure is close to peer review, where authors also act as reviewers of one another. Whether that analogy holds for language models is what the experiment below tests.

## Evaluating LLMs without human labels

The design removes human labels from the loop. Tasks come from the models themselves, filtered by a quality check that the same weighted panel applies. Answers are graded on a 1 to 5 scale with a strict rule against rewarding factual errors. This matters for anyone who needs to compare models in a domain where no public benchmark exists, since the task categories form a list that can be extended by naming a new category and letting the models write the tasks.

Because fresh tasks are drawn every iteration, the benchmark is dynamic in the sense that a dataset cannot be memorized ahead of time. As such, we argue that it resists benchmark contamination better than a fixed test set, a claim that follows from how the tasks are made.

## How well peer rankings agree with MMLU-Pro and GPQA

We ran twelve open-source models through 40 iterations and compared the resulting order against two established benchmarks.

| Benchmark | Multi-judge tau | Multi-judge rho | Single-judge tau | Single-judge rho |
|---|--:|--:|--:|--:|
| MMLU-Pro | 0.64 | 0.78 | 0.45 | 0.61 |
| GPQA | 0.52 | 0.63 | 0.45 | 0.59 |

A Spearman's rho of 0.78 with MMLU-Pro comes from a procedure with no fixed answer key, and the ranking also tracks GPQA, a graduate-level question set, which suggests the signal reaches beyond recall of facts. The multi-judge pipeline beat a variant judged only by gpt-oss-20b, the best of the twelve, on both benchmarks, with the larger gap on MMLU-Pro. The weights of the full panel also settle quickly over the iterations, whereas the single-judge run converges less smoothly.

[Figure: Grouped bars of rank correlation with MMLU-Pro and GPQA. Kendall's tau: multi-judge 0.64 and 0.52, single-judge 0.45 and 0.45. Spearman's rho: multi-judge 0.78 and 0.63, single-judge 0.61 and 0.59.]
Caption: Rank correlation between the AutoBench ranking and each benchmark, for the multi-judge pipeline and the single-judge ablation.

These are rank correlations over twelve models. They show that the peer ranking is consistent with other benchmarks. Whether it matches a human verdict is a separate question. Direct human evaluation remains the reference for qualities such as creativity and subtle reasoning. We describe AutoBench as a complement to it.

## Who can use a dynamic, self-sustaining LLM benchmark

A team that fine-tunes models for a narrow domain could add that domain as a category and track models on it without commissioning a test set. A group that tracks a field of fast-moving open models could rerun the loop whenever a new checkpoint appears. The public AutoBench project already operates at a larger scale than this study, with a third run over 33 models and about 300,000 individual ranks (Kruger, 2025a), and the paper provides the methodology and validation behind it.

The leaderboards reflect a consensus among models. They can inform a decision about which model to deploy. Quality, safety, and alignment need other evidence too, so a leaderboard should not be the only basis.

## Where it goes next

The same loop can take in any model that can write, answer and judge, so the next panels include leading proprietary models alongside open-source ones. Pairing the peer rankings with large-scale human evaluation would then tie the consensus to human judgment directly, on tasks that no fixed benchmark covers.

References:

- Peter Kruger (2025). AutoBench Third Run: Revolutionizing LLM Evaluation with Record-Breaking Scale, Accuracy, and a New Home at autobench.org. Hugging Face Blog. https://huggingface.co/blog/PeterKruger/autobench-run-3
- David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, et al. (2023). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv preprint. https://arxiv.org/abs/2311.12022

BibTeX:

```bibtex
@misc{loi2025autobenchautomatingllmevaluation,
      title={AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment}, 
      author={Dario Loi and Elena Maria Mui{\`a} and Federico Siciliano and Giovanni Trappolini and Vincenzo Cris{\`a} and Peter Kruger and Fabrizio Silvestri},
      year={2025},
      eprint={2510.22593},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2510.22593}, 
}
```

---

From Dario Loi's site (https://dario-loi.github.io/). Index of all papers: https://dario-loi.github.io/llms.txt.
