Instruction-Following Benchmarks: IFEval Is Saturated, IFBench Exposes What's Left

The core question in instruction-following evaluation is simple: did the model do what it was told? Answering it turns out to be hard. Human evaluation is slow and inconsistent. LLM judges introduce bias and circularity. The field's solution has been to focus on verifiable instructions — constraints you can check with a rule rather than a judge. That bet has paid off and created a new problem: models have learned to optimize against the specific constraint vocabulary that evaluators use.

IFEval: the baseline, saturated

IFEval (Google, arXiv 2311.07911) defines 25 types of verifiable instructions across roughly 500 prompts. Verifiable means automatically checkable: write in more than 400 words, include the keyword "luminous" at least three times, return valid JSON, capitalize every sentence. No judge required.

It was a clean benchmark when it launched in 2023. By mid-2026, the top of the leaderboard has converged:

The top-10 spread is 2.9 percentage points. Six of the top 12 slots belong to Qwen3.5 variants. The next non-Qwen model — Claude 3.7 Sonnet — sits at rank 7 with 93.2%.

The Qwen concentration is the giveaway. When one family dominates with this density, the most likely explanation isn't architectural superiority — it's deliberate optimization against IFEval's fixed constraint vocabulary. The 25 constraint types have been stable since 2023, which means any sufficiently large post-training run can target them directly. Models learn to count words and format JSON for these prompts; it doesn't mean they generalize to constraints they haven't seen before.

IFEval is still useful as a sanity check. A model scoring below 85% isn't ready for production instruction-following tasks. But it can no longer tell you which frontier model is actually better at following instructions.

FollowBench: the conjunction problem

FollowBench (ACL 2024) shifts the difficulty axis. Instead of 25 constraint types, it uses five levels of constraint count: each level adds one more constraint to the prompt. The headline metric is Hard Satisfaction Rate (HSR) — a prompt only scores 1 if all constraints are satisfied simultaneously.

HSR is deliberately harsh. A model satisfying 4 of 5 constraints scores 0. This measures constraint conjunction, and the degradation is steep: per-constraint accuracy stays reasonable as constraint count increases; HSR collapses much faster.

The failure mode it exposes is constraint forgetting. Models process constraints sequentially and tend to drop earlier requirements once the prompt stack grows. They look competent on any individual constraint but fail the full conjunction reliably. Example and mixed-constraint prompts show the steepest HSR drops even in GPT-4-class models.

FollowBench isn't saturated at the frontier, but it's not widely tracked on live leaderboards either. Its value is diagnostic: it reveals where in the constraint stack a model starts failing.

InFoBench: partial credit

IFEval and FollowBench both score on binary pass/fail. InFoBench (arXiv 2401.03601, ACL 2024 Findings) takes a different approach with the Decomposed Requirements Following Ratio (DRFR).

Complex instructions get broken into atomic YES/NO sub-checks by GPT-4, each evaluated independently. DRFR is the fraction of sub-requirements met. A 5-part instruction where the model satisfies 4 parts scores 0.8, not 0.

The advantage: DRFR pinpoints which requirement types systematically fail. Content constraints, linguistic style, and example-following each have different error profiles across models. This diagnostic resolution is what IFEval's binary misses.

The limitation: DRFR is more forgiving. A model that reliably misses the same critical constraint every time can still look near-perfect if it satisfies the other nine sub-requirements. DRFR and HSR complement each other — use DRFR for debugging, HSR for deployment readiness.

IFBench: what happens on unseen constraints

IFBench (Allen AI, arXiv 2507.02833, NeurIPS 2025) was built specifically because IFEval's 25 constraint types can be memorized. It introduces 58 novel, diverse, verifiable constraints that are out-of-distribution relative to anything in the training pipeline.

The result is a completely different leaderboard:

The top-3 are reasoning models — Grok 4.3, Grok 4.20, and MiniMax-M3 — scoring 82–83%. Claude models, which score 93–95% on IFEval, drop to 54–59% on IFBench. The 29pp gap between Grok 4.3 and Claude 4.5 Haiku is far larger than anything that separates models on IFEval.

This is the key finding: strong IFEval performance does not transfer to out-of-distribution constraints. Standard instruction-tuning memorizes the known constraint vocabulary; it doesn't build a general capacity to interpret and follow novel requirements.

The reasoning-model advantage suggests a mechanism: explicit thinking at inference time lets models reason about what an unfamiliar constraint actually requires, rather than pattern-match to training examples that match superficially. RLVR (reinforcement learning with verifiable rewards) on constraint satisfaction shows similar gains, per the IFBench paper.

Which benchmark to use

The four benchmarks measure different things and have different saturation states:

IFEval is the floor test: fail it and you're not ready. IFBench is the current frontier discriminator. FollowBench and InFoBench are diagnostic tools — they tell you how a model fails, not just whether it does.

For production decisions: if your use case involves structured, predictable constraints (format rules, JSON, word limits), IFEval and InFoBench are sufficient signals. If you're shipping agents that receive arbitrary natural language constraints from users, IFBench is the evaluation that actually predicts real-world failure.

What IFEval actually looks like

Each IFEval prompt bundles one or more verifiable constraints into a natural-language request. The checker runs a rule against the model's output — no judge needed. Here are four real prompts from the dataset:

Write a short proposal for a new research project that investigates how language evolves over time. I want to make it challenging, so:
1. Do not include any commas in your response.
2. Do not include the letter "c" anywhere in your response.
3. Your response should contain at least 250 words.

Write a poem about how I am missing my classes. The poem must have 4 sections marked with SECTION X. Finish the poem with this exact phrase: "Can I get my money back for the classes I missed?"

Write a story of exactly 2 paragraphs about a man who wakes up one day and realizes that he's inside a video game. Separate the paragraphs with the markdown divider: ***

Write a 300+ word summary of the wikipedia page "https://en.wikipedia.org/wiki/Raymond_III,_Count_of_Tripoli". Do not use any commas and highlight at least 3 sections that has titles in markdown format, for example *highlighted section part 1*, *highlighted section part 2*, *highlighted section part 3*.