Harness tax
The same benchmark, measured differently
The same benchmark, run under different harnesses or protocols, disagrees about the same models - on the agentic benchmarks by far more than on the tool-free ones. Measured scores only: no estimate enters this page.
Pooled mean |delta| by tier - agentic: 6.9 pp · knowledge_tool_free: 2.3 pp · protocol_layer: 8.5 pp; verified families only, the agentic vs tool-free ratio is 3.7x.
| Pair | Models | Mean |Δ| | Kendall τ | Direction |
|---|---|---|---|---|
| chartography (published) vs chartographywithtools (published) untiered · protocol_variant · weak_alignment · candidate family · low overlap · chartography | 3 | 35.5 pp | 0.33 | 0 up / 3 down |
| aa-automation-bench (artificial-analysis) vs automationbench (published) untiered · cross_harness · weak_alignment · candidate family · automationbench | 13 | 19.7 pp | 0.15 | 13 up / 0 down |
| riemannbench (published) vs riemannbenchwithtools (published) untiered · protocol_variant · weak_alignment · candidate family · low overlap · riemannbench | 1 | 19.0 pp | — | 0 up / 1 down |
| terminalbench21 (published) vs vals-terminal-bench21 (vals-ai) agentic · cross_harness · verified · terminal-bench-2-1 | 29 | 12.1 pp | 0.52 | 29 up / 0 down |
| hlenotools (published) vs hlewithtools (published) protocol_layer · protocol_variant · verified · hle-protocol-variants | 11 | 10.0 pp | 0.81 | 0 up / 11 down |
| vals-terminal-bench21 (vals-ai) vs aa-terminal-bench21 (artificial-analysis) agentic · cross_harness · verified · terminal-bench-2-1 | 11 | 9.8 pp | 0.88 | 0 up / 11 down |
| hle (published) vs aa-hle (artificial-analysis) protocol_layer · cross_harness · needs_audit · hle | 48 | 8.2 pp | 0.58 | 28 up / 18 down |
| livecodebench-pro (published) vs valslivecodebench (vals-ai) agentic · cross_harness · weak_alignment · candidate family · low overlap · livecodebench | 2 | 7.9 pp | 1.00 | 0 up / 2 down |
| valslivecodebench (vals-ai) vs aa-live-code-bench (artificial-analysis) agentic · cross_harness · weak_alignment · candidate family · low overlap · livecodebench | 1 | 7.2 pp | — | 0 up / 1 down |
| arxivmathaugust2026 (published) vs arxivmathaugust2026withtools (published) untiered · protocol_variant · weak_alignment · candidate family · low overlap · arxivmathaugust2026 | 2 | 7.1 pp | 1.00 | 0 up / 2 down |
| charxiv (published) vs charxivnotools (published) untiered · protocol_variant · weak_alignment · candidate family · charxiv | 12 | 6.5 pp | 0.66 | 12 up / 0 down |
| swe-bench-verified (published) vs valsswebench (vals-ai) agentic · cross_harness · needs_audit · swe-bench-verified | 24 | 6.3 pp | 0.68 | 17 up / 4 down |
| livecodebench-v6 (published) vs livecodebench-pro (published) agentic · protocol_variant · weak_alignment · candidate family · low overlap · livecodebench | 3 | 6.1 pp | 1.00 | 3 up / 0 down |
| livecodebench (published) vs aa-live-code-bench (artificial-analysis) agentic · cross_harness · weak_alignment · candidate family · low overlap · livecodebench | 1 | 4.5 pp | — | 0 up / 1 down |
| ifbench (published) vs aa-if-bench (artificial-analysis) knowledge_tool_free · cross_harness · verified · ifbench | 11 | 4.2 pp | 0.77 | 6 up / 3 down |
| terminal-bench-4 (published) vs aa-terminal-bench4 (artificial-analysis) agentic · cross_harness · verified · terminal-bench-4-0 | 11 | 3.9 pp | 0.82 | 8 up / 2 down |
| livecodebench (published) vs valslivecodebench (vals-ai) agentic · cross_harness · weak_alignment · candidate family · low overlap · livecodebench | 2 | 3.6 pp | 1.00 | 2 up / 0 down |
| livecodebench-v6 (published) vs valslivecodebench (vals-ai) agentic · cross_harness · weak_alignment · candidate family · low overlap · livecodebench | 4 | 3.3 pp | -0.18 | 4 up / 0 down |
| terminalbench21 (published) vs aa-terminal-bench21 (artificial-analysis) agentic · cross_harness · verified · terminal-bench-2-1 | 10 | 3.3 pp | 0.90 | 7 up / 1 down |
| vals-gpqa-diamond (vals-ai) vs aa-gpqa-diamond (artificial-analysis) knowledge_tool_free · cross_harness · verified · gpqa-diamond | 47 | 2.9 pp | 0.77 | 15 up / 31 down |
| mmlu-pro (published) vs mmlu-pro-arcee (published) knowledge_tool_free · protocol_variant · needs_audit · low overlap · mmlu-pro | 3 | 2.4 pp | -0.33 | 0 up / 2 down |
| gdppdf (published) vs gdppdfwithtools (published) untiered · protocol_variant · weak_alignment · candidate family · low overlap · gdppdf | 1 | 2.1 pp | — | 0 up / 1 down |
| scicode (published) vs aa-sci-code (artificial-analysis) agentic · cross_harness · needs_audit · scicode | 20 | 2.0 pp | 0.77 | 6 up / 11 down |
| mmmu-pro (published) vs aa-mmmu-pro (artificial-analysis) knowledge_tool_free · cross_harness · verified · mmmu-pro | 32 | 2.0 pp | 0.70 | 16 up / 11 down |
| gpqa-diamond (published) vs aa-gpqa-diamond (artificial-analysis) knowledge_tool_free · cross_harness · verified · gpqa-diamond | 40 | 2.0 pp | 0.81 | 21 up / 16 down |
| mmlu-pro (published) vs vals-mmlu-pro (vals-ai) knowledge_tool_free · cross_harness · needs_audit · mmlu-pro | 7 | 1.8 pp | 0.52 | 5 up / 1 down |
| gpqa-diamond (published) vs vals-gpqa-diamond (vals-ai) knowledge_tool_free · cross_harness · verified · gpqa-diamond | 23 | 1.4 pp | 0.69 | 15 up / 8 down |
| arxivmathjune2026 (published) vs arxivmathjune2026withtools (published) untiered · protocol_variant · weak_alignment · candidate family · low overlap · arxivmathjune2026 | 1 | 0.5 pp | — | 0 up / 1 down |
| vals-mmlu-pro (vals-ai) vs mmlu-pro-arcee (published) knowledge_tool_free · cross_harness · needs_audit · low overlap · mmlu-pro | 1 | 0.4 pp | — | 0 up / 1 down |
| benchcadvision2code (published) vs benchcadvision2codewithtools (published) untiered · protocol_variant · weak_alignment · candidate family · low overlap · benchcadvision2code | 3 | 0.3 pp | 1.00 | 0 up / 3 down |
| aime2025 (published) vs aime2025arcee (published) untiered · protocol_variant · weak_alignment · candidate family · low overlap · aime2025 | 1 | 0.2 pp | — | 0 up / 1 down |
| aa-aime2025 (artificial-analysis) vs aime2025 (published) untiered · cross_harness · weak_alignment · candidate family · low overlap · aime2025 | 0 | — | — | — |
| aa-aime2025 (artificial-analysis) vs aime2025arcee (published) untiered · cross_harness · weak_alignment · candidate family · low overlap · aime2025 | 0 | — | — | — |
| aa-gdp-pdf (artificial-analysis) vs gdppdf (published) untiered · cross_harness · weak_alignment · candidate family · low overlap · gdppdf | 0 | — | — | — |
| aa-gdp-pdf (artificial-analysis) vs gdppdfwithtools (published) untiered · cross_harness · weak_alignment · candidate family · low overlap · gdppdf | 0 | — | — | — |
| livecodebench (published) vs livecodebench-v6 (published) agentic · protocol_variant · weak_alignment · candidate family · low overlap · livecodebench | 0 | — | — | — |
| livecodebench (published) vs livecodebench-pro (published) agentic · protocol_variant · weak_alignment · candidate family · low overlap · livecodebench | 0 | — | — | — |
| livecodebench-v6 (published) vs aa-live-code-bench (artificial-analysis) agentic · cross_harness · weak_alignment · candidate family · low overlap · livecodebench | 0 | — | — | — |
| livecodebench-pro (published) vs aa-live-code-bench (artificial-analysis) agentic · cross_harness · weak_alignment · candidate family · low overlap · livecodebench | 0 | — | — | — |
| aa-math500 (artificial-analysis) vs math-500 (published) untiered · cross_harness · weak_alignment · candidate family · low overlap · math500 | 0 | — | — | — |
| terminal-bench-3 (published) vs terminal-bench3 (published) agentic · protocol_variant · verified · candidate family · low overlap · terminal-bench-3 | 0 | — | — | — |
| terminal-bench-science (published) vs terminal-bench-science-6x-verifier (published) protocol_layer · protocol_variant · verified · candidate family · low overlap · terminal-bench-science-variants | 0 | — | — | — |
Measured scores only, never estimates; every per-model delta keeps both scores' retrieval dates and source URLs in the API. Families whose item sets are not verified are flagged and stay out of the headline aggregates; low-overlap pairs are greyed on the interactive page; 23 per-model deltas beyond 20 pp sit in the audit queue.