Chat & Arena Benchmarks: Human Votes Scale, MT-Bench Doesn't

Verifiable instruction benchmarks (covered in post 1 of this series) have a built-in ceiling: once models satisfy the constraint set reliably, there's nowhere to go. Chat benchmarks try a different approach — measure what humans actually prefer, in the kind of open-ended conversation that fills most real LLM usage. This solves the ceiling problem but creates its own: preference evaluation is expensive, slow, noisy, and hard to make reproducible.

The four benchmarks in this space have taken different bets. Some have saturated. One keeps getting more useful as it grows.

LMArena: Elo from the crowd

LMArena (formerly LMSYS Chatbot Arena, rebranded to arena.ai in January 2026 after raising a $150M Series A) is the oldest and largest human preference platform for LLMs. The setup: two models respond anonymously to the same prompt; the human picks the better response. Results feed a Bradley-Terry Elo system, the same model that ranks chess players.

The current leaderboard (July 2026, 192 models, millions of votes):

Three Anthropic models sit in the top three. The top-10 span is 32 Elo. That sounds tight — and at low vote counts it would be statistically meaningless — but with millions of total comparisons and hundreds of thousands of votes per top model, a 32-Elo gap is real signal. It means the higher-ranked model wins roughly 54% of pairwise comparisons against the lower-ranked one.

The full leaderboard spans 528 Elo (from Claude Opus 4.6 at 1500 to ChatGLM3-6B at 972), which shows the system still has plenty of room. It can rank frontier models at the top and legacy small models at the bottom without hitting any ceiling.

Two things make Arena noisy in practice:

Vote heterogeneity. Users vote on prompts they care about. Someone testing code generation votes differently than someone evaluating creative writing. Category-specific leaderboards (Arena has them for coding, math, creative writing) show large rank swaps between categories. The overall leaderboard is a weighted average of those distributions.

Confidence intervals compress slowly. A model with 2,000 votes has a CI of roughly ±15 Elo; with 50,000 votes, ±5 Elo. Top models accumulate votes fast, but new models enter with wide CIs. When reading the leaderboard, ranks within ±10 Elo of each other are not statistically resolved.

MT-Bench: the original multi-turn judge

MT-Bench (LMSYS, 2023) predates Arena as an automated alternative. It uses 80 two-turn questions across 8 categories — writing, roleplay, extraction, reasoning, math, coding, STEM, humanities — and asks GPT-4 to score each response on a 1–10 scale.

MT-Bench introduced the idea of LLM-as-judge at scale, and it was discriminating when it launched. GPT-4 scored 8.99, Claude 2 scored 8.06, and the gap was clear. By mid-2026, the benchmark tracks only 12 models on llm-stats.com, and the top model (Hermes 3 70B) sits at 8.99 — essentially tied with where GPT-4 was three years ago, because frontier models have long since passed that level.

The saturation is fundamental: GPT-4's scoring range compresses near 9–10. When multiple models all score 9+, the differences are noise from GPT-4's inconsistent grading, not real quality separation. MT-Bench is now most useful for evaluating sub-70B open-source models that still score 7–8.5, where the scale has room.

AlpacaEval 2.0: length-controlled win rates

AlpacaEval 2.0 (Stanford, 2024) evaluates 805 diverse instructions by asking GPT-4 Turbo to judge whether the tested model's response is better than a reference model (GPT-4 Turbo). The key innovation over AlpacaEval 1.0: length-controlled (LC) win rate regresses out response length from the preference score.

The length bias it addresses is real. Standard LLM judges prefer longer responses as higher quality even when the extra content is filler. LC win rate controls for this statistically, producing a score that represents quality independent of verbosity.

When it launched, LC win rate was discriminating across a broad field. By mid-2026, frontier models — Claude Opus 4.7, GPT-5.x, Gemini 3 series, Llama 4 — are all scoring above 95% LC win rate. The top tier has compressed to within 5pp, and the benchmark has soft-saturated at the frontier. The community has largely moved to Arena-Hard-Auto and WildBench for headline win-rate comparisons.

AlpacaEval 2.0 still has value below the frontier. The 805 prompts are diverse enough that a 75% LC win rate and an 85% LC win rate represent meaningfully different models. But it cannot rank the top five current models against each other.

WildBench: real user prompts

WildBench (Allen AI, 2024) takes the source-data approach: 1,000+ tasks drawn from real ChatGPT user conversations, covering reasoning, planning, coding, and creative writing as they actually appear in practice — not as benchmark designers assume they appear.

Two metrics: WB-Reward (pairwise preference between two models, GPT-4-judged) and WB-Score (absolute quality rating against task-specific checklists). The checklists are generated per-task and constrain the judge to evaluate against concrete requirements rather than vague quality impressions.

The current leaderboard leader (Mistral Large 3) scores 68.5% on WB-Score. The top-10 spread is roughly 10pp — much more room than MT-Bench or AlpacaEval 2.0 at the frontier. WildBench remains discriminating because the real-user task distribution keeps producing prompts that expose genuine model differences.

The core advantage of WildBench over synthetic benchmarks is contamination resistance. Models trained before the dataset was released cannot have been optimized against its specific prompts. And because the prompts are real, they include the messy, underspecified requests that synthetic benchmark designers tend to clean up or omit.

Arena-Hard-Auto: harder, cheaper

Arena-Hard-Auto (also from LMSYS) is the automated alternative to human voting. It uses 500 challenging, diverse queries sourced from Chatbot Arena conversations and judges head-to-head responses using a stronger model (GPT-4o or equivalent). The result is an automated Elo-like ranking that correlates well with actual Arena results.

Arena-Hard v2 (June 2026) expanded to 750 queries with better category coverage and updated reference comparisons. It provides a useful middle ground: faster and cheaper than running a full Arena evaluation, more recent and challenging than MT-Bench. Arena-Hard-Auto v2 currently shows 16 evaluated models with a spread that makes it discriminating at the frontier.

Saturation across the four benchmarks

The pattern: automated benchmarks with fixed rubrics or fixed reference points saturate faster than benchmarks anchored to human preferences or real user data. MT-Bench's GPT-4 judge ran out of room by late 2024. AlpacaEval 2.0's LC reference ran out of room at the frontier in 2025. LMArena doesn't saturate by design — every new model competes against everything else, and the Elo spread grows as models diverge at the extremes.

WildBench stays useful longer than AlpacaEval for a structural reason: real user prompts are a moving target. Each new wave of user data reflects what people are actually asking current-generation models to do, which tends to be harder than what was available a year earlier.

What this means for model selection

If you're choosing between top-tier frontier models for a general chat product:

• LMArena overall Elo is the best available signal for general preference.
• Category-specific Arena scores (coding, creative, etc.) are better if your use case is narrow.
• WildBench provides a useful second opinion based on real-world task diversity.
• MT-Bench and AlpacaEval 2.0 are no longer useful for frontier model differentiation — treat them as historical reference.

If you're evaluating smaller models (7B–70B open-source):

• MT-Bench still discriminates in the 7–8.5 range.
• AlpacaEval 2.0 still has headroom below ~80% LC win rate.
• Arena scores from specialized hosting setups give the most reliable signal, but require vote volume that takes time to accumulate.

What MT-Bench actually looks like

MT-Bench uses 80 two-turn questions across 8 categories. Each question is scored by GPT-4 on a 1–10 scale. The multi-turn format is what distinguishes it from single-shot evals — the second turn often requires the model to apply or transform what it said in turn 1. Here are four real questions spanning four different categories:

Compose an engaging travel blog post about a recent trip to Hawaii, highlighting cultural experiences and must-see attractions.

Rewrite your previous response. Start every sentence with the letter A.

Imagine you are participating in a race with a group of people. If you have just overtaken the second person, what's your current position? Where is the person you just overtook?

If the "second person" is changed to "last person" in the above question, what would the answer be?

The vertices of a triangle are at points (0, 0), (-1, 1), and (3, 3). What is the area of the triangle?

What's area of the circle circumscribing the triangle?

Develop a Python program that reads all the text files under a directory and returns top-5 words with the most number of occurrences.

Can you parallelize it?