Better AI scoreboards start by showing how the test was run
AISI and EvalEval are sharing evaluation records that make model comparisons easier to interrogate.
On September 22, EvalEval and the UK AI Security Institute described a release of evaluation results with configuration and context through Evaluation Cards. The shared Every Eval Ever schema provides a common format for storing and interpreting records from different evaluation systems. [1] [2] [3]
The release includes five benchmarks from the main experiment in the paper How Inference Compute Shapes Frontier LLM Evaluation: HealthBench, FrontierMath, Humanity’s Last Exam, SWE-Bench Pro and Terminal-Bench 2.0. It also includes related cyber evaluations with a partially different set of models. [1] [2] [3]
The underlying research examines how results depend on inference-time compute and evaluation protocol. A score obtained with repeated attempts and feedback does not describe the same conditions as a score from one unaided attempt. Recording those conditions helps readers understand what a comparison actually measures. [1] [2] [3]
Why it matters
That matters to anyone choosing software, even if they never read a benchmark paper. A school, small business or research team needs a tool that works within its own time, money and review constraints. A leaderboard number without the setup can conceal a mismatch with those needs.
Open records are not a certificate of safety or a guarantee of reproducibility. The tasks can still miss important failures, and a reader may not have the resources to rerun an expensive experiment. But a visible configuration gives criticism a concrete starting point.
Our practical question for the next model comparison is simple: can someone explain the tools, attempt budget, feedback and scoring behind it? Progress includes improving the evidence used to judge progress. Sharing those details is useful public-interest infrastructure, even when it comes without a dramatic demo.
Limits of this reporting
Reporting infrastructure does not establish model safety, universal task reliability or reproducibility at zero cost.
Sources & evidence
- How UK AISI and EvalEval Are Making Benchmark Results Reproducible — EvalEval and UK AI Security Institute. Published 2026-09-22; accessed 2026-10-05.
- How Inference Compute Shapes Frontier LLM Evaluation — Research authors via arXiv. Published 2026-06; accessed 2026-10-05.
- Every Eval Ever schema and database — EvalEval. Published date not stated; accessed 2026-10-05.
Source reporting and our analysis are separated in the text. Editorial policy.