Method
How the gaps are filled, and when not to trust it
benchgap is an LLM benchmark leaderboard that fills in the missing scores. Most models are only ever run on a handful of benchmarks, so benchgap calibrates benchmarks against each other on the models measured on both, then estimates each missing score with its cross-validated error and a confidence level. Measured and estimated scores are always marked apart.
- Measured scores come from public evaluation leaderboards and are never altered; each benchmark is calibrated only against benchmarks of the same capability.
- For every ordered pair of benchmarks with at least 5 models measured on both, 6 monotone curves are fitted and the rising one with the lowest leave-one-out cross-validated error is kept (a falling fit is an overfit to too few models, never a calibration). That error, in percentage points, is the ± shown with every estimate.
- A pair keeps no calibration unless its best curve reaches R² ≥ 0.3 and an error of at most 15 pp; such gaps stay empty.
- A missing score is estimated from the best calibration out of a benchmark the model was measured on, if its score there lies within one training-range width of the scores the calibration was fitted on; further out the gap stays empty. Estimates are never used to make further estimates.
- Several same-capability benchmarks may be combined into a multivariate mapping: an elastic net with non-negative coefficients whose lasso zeroes the useless sources or a greedy forward search finds the features, and then a linear fit and a multivariate Michaelis–Menten curve compete. It is stored only if it passes the gate and beats the target's best single calibration, and preferred for a model measured on all of its sources - a model missing one source falls back to the univariate path.
- Cross-domain predictability and the multivariate view are analyses only: no estimate crosses a capability.
- Confidence: high for an error up to 5 pp, medium up to 10 pp, low above; extrapolation, a fit on fewer than 8 models or R² below 0.5 each lower it by one level.
- The harness tax (see /harness-tax) measures how much harnesses disagree about the same models on the same benchmark, from measured scores only; it never feeds the estimates, and it is why each of them holds for its source harness's evaluation setup only.
- Estimates are predictions, not measurements, and hold for the source leaderboard's evaluation setup only.
- The whole pipeline is validated end to end: masking measured scores and refitting everything, estimates fill about 70% of masked cells at about 4.7 pp mean absolute error (23% lower than a matrix-completion baseline on the same cells); the confidence levels are correctly ordered (3.1 / 5.1 / 7.5 pp realized MAE for high/medium/low), though the ± labels understate the realized RMSE by roughly 40–70%; reweighted to the published mix, a published estimate should be expected to carry about 5.9 pp MAE (9.1 pp RMSE).