BENCHMARK EVIDENCE / V1
First-party results are useful signals—not a leaderboard. Every score keeps its model scope, reported settings, source, and missing run context attached.
Comparable means same artifact and methodology fingerprint. Everything else remains evidence, not rank.
REPORTED EVIDENCE STREAM
These values come from official model cards and technical reports. None is represented as independently reproduced, cryptographically bound to the evaluated weights, or safe for cross-model ranking.
Open the evidence APIKimi K3
Diamond
reasoning_effort=maxtemperature=1top_p=0.95Kimi K3
2.1
reasoning_effort=maxtemperature=1top_p=1harness=Kimi CodeKimi K3
Version not reported
reasoning_effort=maxtemperature=1top_p=1context_compaction_trigger_tokens=300000DeepSeek V4 Pro
Diamond
reasoning_mode=Think Maxtemperature=1context_tokens=384000system_prompt_variant=Think Max special system promptDeepSeek V4 Pro
v6
reasoning_mode=Think Maxsystem_prompt_variant=Think Max special system promptDeepSeek V4 Pro
Verified
reasoning_mode=Think Maxharness=DeepSeek internal evaluation frameworktooling=bash and file-edit toolsmax_interaction_steps=500context_tokens=512000GLM-5.2
Diamond
temperature=1top_p=0.95max_new_tokens=163840GLM-5.2
Version not reported
framework=OpenHandsprompt_variant=tailored instruction prompttemperature=1top_p=1max_new_tokens=32000context_tokens=400000GLM-5.2
500-task public set
thinking_mode=truetask_count=500timeout_seconds=600judge_model=Gemini-3.0-ProGemma 4 26B A4B IT
2026 · no tools
model_stage=instruction-tunedthinking_mode=truetool_use=falseGemma 4 26B A4B IT
v6
model_stage=instruction-tunedthinking_mode=trueGemma 4 26B A4B IT
Diamond
model_stage=instruction-tunedthinking_mode=trueQwen3-30B-A3B
Diamond
thinking_mode=truetemperature=0.6top_p=0.95top_k=20max_new_tokens=32768samples_per_query=10Qwen3-30B-A3B
2025 · Parts I and II
thinking_mode=truetemperature=0.6top_p=0.95top_k=20max_new_tokens=38912question_count=30samples_per_query=64Qwen3-30B-A3B
v5 · 2024-10 through 2025-02
thinking_mode=truetemperature=0.6top_p=0.95top_k=20max_new_tokens=32768benchmark_window=2024-10 through 2025-02prompt_variant=official prompt with the program-only restriction removedTHE RESOLUTION QUESTION
Scoring a benchmark is a sequence of Bernoulli trials, so a reported accuracy carries a standard error whether or not anyone prints it. On a small instrument that error is larger than the gaps leaderboards rank by.
Closed-form arithmetic over the values you enter. No model scores are supplied, assumed or stored.
NOT RESOLVABLE AT THIS SIZE
A 3.00 point gap on GPQA Diamond carries a 95% margin of ±9.84 points. The interval on the difference runs from -6.84 to 12.84 points, which contains zero. The two scores are indistinguishable on this instrument.
| Observed difference | 3.00 pp | Model A minus Model B |
|---|---|---|
| Standard error, unpaired | 5.02 pp | √(SE² A + SE² B), treating the runs as independent samples |
| Standard error, paired | 5.02 pp | √(SE² A + SE² B − 2ρ·SE A·SE B) at ρ = 0.00 |
| Design effect | 1.00× | 1 + (m − 1)·ICC at m = 1, ICC = 0.00 |
| Effective questions | 198 | 198 nominal ÷ design effect |
| Smallest resolvable gap | 9.84 pp | Anything below this cannot be distinguished from noise here |
| Questions needed, unpaired | 4,361 | (zα/2 + zβ)² · 2p(1−p) ÷ δ² at 80% power |
| Questions needed, paired | 4,361 | Same, with variance reduced by (1 − ρ) |
| Questions needed, clustered | 4,361 | Paired requirement × design effect |
GPQA Diamond is short by 4,163 questions for this comparison. Reporting the ordering anyway is reporting noise.
The hardest GPQA subset, and the one leaderboards quote most often. Source ↗
ρ is the correlation between the two models’ per-question outcomes. It is left at zero by default because it can only be measured from per-question outputs, which almost no leaderboard publishes; raising it without that measurement produces an interval narrower than the evidence supports. Clustering applies when questions arrive in groups — several per passage, per repository, per record — because those questions are not independent draws.
The default view is the case leaderboards publish constantly: a three-point gap on GPQA Diamond, which has 198 questions. The 95% margin on that difference is roughly ten points. The gap would have to more than triple before the benchmark could tell the two models apart, and no site reporting that ordering says so.
Two levers make comparisons cheaper rather than louder. Pairing — scoring both models on the same questions and comparing per-question differences — removes the variance the models share, and it is why a comparison that needs several thousand questions when treated as two independent means can need roughly a thousand when paired. Clustering pushes the other way: when questions arrive in groups, the effective sample is smaller than the question count.
Method after Miller, “Adding Error Bars to Evals” (Anthropic, 2024): arxiv.org/abs/2411.00640 ↗
DETERMINISTIC INGESTION
The v0.1 importer hashes aggregate bytes, preserves metric/filter pairs and explicit unknowns, and never executes model or task code. Staging output is not publication eligibility.
MOEMODELS / EVIDENCE IMPORT$ moemodels ingest lm-eval results.json \
--source-url https://example.org/results.json \
--retrieved-at 2026-08-03 \
--model qwen3-30b-a3b \
--jsonThe comparison gate requires a matching fingerprint. A valid but incomplete import remains useful—and visibly excluded from rankings until the consequential fields exist.
01checkpoint revision02raw artifact sha25603harness commit04task hash + version05dataset revision06chat template hash07system prompt hash08reasoning settings09sampling + seeds10sample counts + limit11runtime + precision12hardware topologyThat is deliberate. We will publish the first normalized result only when its raw artifact, exact model revision, harness identity, task configuration, sample counts, and known gaps survive validation.
CONTRIBUTION STANDARD
The open schema is built for reviewable results, independent reproductions, and corrections that preserve history.
Open DeployBench