Fewer Human Evals, Better Model Rankings

Active arena evals rank model versions with far fewer human reviews by choosing the head-to-head comparison most likely to improve the leaderboard.

That matters because exhaustive arena grading gets expensive quickly. If you have n models and m benchmark tasks, the full grid asks a human to compare every model pair on every task:

def brute_force_comparisons(num_models: int, num_tasks: int) -> int:
    return num_models * (num_models - 1) // 2 * num_tasks


brute_force_comparisons(10, 20)  # 900

Nine hundred human judgments might be useful for a final benchmark paper. It is a lot for day-to-day model iteration, where the real question is simpler: which model is clearly better, which one is worse, and which models are too close to separate without more evidence?

The goal is not to observe the whole comparison graph. The goal is to spend human attention where it buys the most information.

The Basic Arena Unit

Each arena judgment compares two model outputs on one task:

comparison = {
    "model_a": "agent_042",
    "model_b": "agent_039",
    "task_id": "task_yc_founders",
    "grade": 3,
}

The grade is on a -5 to +5 scale:

• -5 means model B is much better.
• 0 means the outputs are tied or too close to call.
• +5 means model A is much better.

The nice thing about this scale is that it stores both direction and margin. A tiny preference should move ratings less than a clear win.

def grade_to_target(grade: int) -> float:
    eps = 0.02
    target = (grade + 5) / 10
    return min(1 - eps, max(eps, target))

This turns the human grade into a soft preference target. A blowout gets close to 0 or 1. A tie lands near 0.5.

What We Can Safely Assume

The useful assumptions are modest:

• Transitivity is helpful, but not perfect.
• We need approximate ordering, not fake precision.
• Some tasks separate models better than others.
• Big wins usually contain more signal than tiny wins.
• Weak models should stop consuming comparisons once the evidence is clear.
• Models that are too close should be grouped together instead of forced into arbitrary ranks.

Approximate transitivity is the main savings:

# Evidence, not law:
# A beats B
# B beats C
# therefore A probably beats C

This should reduce work, but it should not erase uncertainty. A model can be strong on extraction, weak on code, and tied on writing. The system should remember that tasks are not interchangeable.

Score The Models

A simple Bradley-Terry model is enough for the first serious version. Each model has a latent strength score, and a pairwise comparison estimates the probability that one model beats another.

from math import exp


def sigmoid(x: float) -> float:
    return 1 / (1 + exp(-x))


def expected_preference(beta_a: float, beta_b: float, task_discrimination: float) -> float:
    return sigmoid(task_discrimination * (beta_a - beta_b))

Each model stores:

model_state = {
    "beta": 0.0,
    "sigma": 1.0,
    "eval_count": 0,
}

Each task stores:

task_state = {
    "discrimination": 1.0,
    "noise": 1.0,
    "eval_count": 0,
}

beta is the model's strength. sigma is uncertainty. Task discrimination measures whether a task actually separates models. A task where every model fails equally may be hard, but it is not useful for ranking.

The Math Behind The Scores

The friendly version is: every model gets a hidden strength, every task gets a discrimination weight, and every human grade is a noisy measurement of the gap between two models.

Let model $i$ have latent strength:

For model $a$ against model $b$ on task $t$, define the latent gap:

Task $t$ has discrimination $d_t > 0$. The predicted preference for $a$ over $b$ is:

The human grade is mapped to:

The assumption is deliberately gentle:

So we are not assuming humans are perfect. We are assuming the grade is a noisy signal that points in the right direction on average:

We fit the model with weighted Bradley-Terry loss:

The comparison weight is higher when the task is discriminative and lower when the task is noisy:

where $\eta_t$ is the estimated noise for task $t$.

Because only differences in $\beta$ matter, we center the scores after fitting:

That normalization does not change any predicted matchup. It just keeps the coordinate system tidy.

Why The Active Rule Is Good

The active arena picks comparisons by information gain. Here is the core reason close comparisons are valuable.

For one comparison, let $x$ be a vector with $+1$ at model $a$, $-1$ at model $b$, and $0$ everywhere else. Then:

The gradient of the log-likelihood is proportional to:

The Fisher information from this comparison is:

The scalar part is:

The term $p(1-p)$ is maximized at $p = 1/2$. So the system learns most when it compares models whose outcome is uncertain.

Proof.

so $p=1/2$ is the maximum. That is the whole reason the algorithm prefers close, uncertain comparisons over obvious blowouts.

Update Scores Online

For interactive grading, update scores after every human judgment. This makes the leaderboard feel live without waiting for a full refit.

from math import sqrt


def update_from_grade(
    beta: dict[str, float],
    sigma: dict[str, float],
    task_discrimination: dict[str, float],
    task_noise: dict[str, float],
    model_a: str,
    model_b: str,
    task_id: str,
    grade: int,
    learning_rate: float = 0.05,
) -> None:
    y = grade_to_target(grade)

    disc = task_discrimination[task_id]
    noise = max(task_noise[task_id], 0.25)

    p = expected_preference(beta[model_a], beta[model_b], disc)
    error = y - p

    task_weight = disc / noise
    step = learning_rate * task_weight * disc * error

    beta[model_a] += step
    beta[model_b] -= step

    info = task_weight * (disc ** 2) * p * (1 - p)
    sigma[model_a] = 1 / sqrt((1 / sigma[model_a] ** 2) + info)
    sigma[model_b] = 1 / sqrt((1 / sigma[model_b] ** 2) + info)

The online update is fast. Periodically, refit from the full match history with a weighted Bradley-Terry loss so the live updates do not drift over time.

def refit_global_scores(comparisons: list[dict]) -> None:
    """
    Fit beta by minimizing weighted soft-label Bradley-Terry loss.
    Use grade_to_target(grade) as the target.
    Weight each comparison by task discrimination and task noise.
    """
    raise NotImplementedError

Pick The Next Best Comparison

The active arena should not ask a random match. It should ask the match that is most likely to change what we believe.

A candidate is a triple:

candidate = ("model_a", "model_b", "task_id")

The value of that candidate is high when:

• the expected outcome is uncertain,
• both models still have high rating uncertainty,
• the task is reliable,
• the model pair has not already been over-sampled,
• the task has not already received enough judgments.

def candidate_value(
    beta: dict[str, float],
    sigma: dict[str, float],
    task_discrimination: dict[str, float],
    task_noise: dict[str, float],
    eval_count: dict[str, int],
    task_eval_count: dict[str, int],
    seen_pair_task: dict[tuple[str, str, str], int],
    model_a: str,
    model_b: str,
    task_id: str,
) -> float:
    disc = task_discrimination[task_id]
    noise = max(task_noise[task_id], 0.25)

    p = expected_preference(beta[model_a], beta[model_b], disc)
    outcome_uncertainty = p * (1 - p)
    rating_uncertainty = sigma[model_a] ** 2 + sigma[model_b] ** 2
    task_quality = disc / noise

    pair_key = tuple(sorted((model_a, model_b)) + [task_id])
    repeat_penalty = 1 / sqrt(1 + seen_pair_task.get(pair_key, 0))

    model_coverage = (
        1 / sqrt(1 + eval_count[model_a])
        + 1 / sqrt(1 + eval_count[model_b])
    )
    task_coverage = 1 / sqrt(1 + task_eval_count[task_id])

    return (
        outcome_uncertainty
        * rating_uncertainty
        * task_quality
        * repeat_penalty
        * (1 + 0.15 * model_coverage)
        * (1 + 0.15 * task_coverage)
    )

The key term is:

p * (1 - p)

It is largest when p = 0.5. Close comparisons are informative. Obvious blowouts are usually not.

One-Step Optimality

The heuristic above is the version I would ship first. The more mathematical version keeps a covariance matrix $V$ over the model scores. Large covariance means we are still uncertain.

For a candidate comparison with vector $x$, the information matrix update is:

where:

Using the Sherman-Morrison identity:

we get:

Now pick any model gap we care about:

For example, $c$ might be $+1$ for model A and $-1$ for model B. Its current variance is:

After adding candidate comparison $x$, the variance reduction is:

So if we care about a set of tier-boundary gaps $\mathcal{C}$, the one-step best comparison is:

That is the proof-backed version of "ask the next most useful question." It is not claiming the greedy policy is globally perfect across every future batch. It is claiming something narrower and more useful: under the current uncertainty estimate, this comparison gives the largest immediate reduction in the uncertainty we care about.

In Python-shaped pseudocode:

def variance_reduction(V, x, c, info):
    numerator = info * float(c.T @ V @ x) ** 2
    denominator = 1 + info * float(x.T @ V @ x)
    return numerator / denominator


def information_gain(V, x, info, important_contrasts):
    return sum(
        variance_reduction(V, x, c, info)
        for c in important_contrasts
    )

Rank By Tiers, Not Fake Precision

Exact rankings waste effort near ties. If two models are within a practical margin, group them together until the evidence says otherwise.

PRACTICAL_MARGIN_ELO = 50

Convert Bradley-Terry scores into an Elo-like display score:

from math import log


def beta_to_elo(beta_value: float, base: float = 1000) -> float:
    scale = 400 / log(10)
    return base + scale * beta_value

Then build tiers from confidence intervals:

def should_separate(elo_a: float, sigma_a: float, elo_b: float, sigma_b: float) -> bool:
    lower_a = elo_a - 1.96 * sigma_a
    upper_b = elo_b + 1.96 * sigma_b
    return lower_a > upper_b + PRACTICAL_MARGIN_ELO

This gives the leaderboard a more honest shape:

tiers = [
    ["agent_042", "agent_041"],
    ["agent_039", "agent_038", "agent_036"],
    ["agent_030"],
]

If two models are indistinguishable in practice, the product should say that. It is better than pretending rank 3 and rank 4 are meaningfully different because one has two more Elo points.

A Conservative Tier Guarantee

Here is the guarantee I would want the product to stand behind.

Let $\hat{\beta}_i$ be the fitted score for model $i$, and let $r_i$ be its confidence radius. Assume the intervals are simultaneous, meaning:

Define:

Only put model $a$ in a strictly higher tier than model $b$ when:

where $\delta$ is the practical margin.

Theorem. With probability at least $1-\alpha$, every separated tier boundary is correct up to margin $\delta$.

Proof. On the event that all confidence intervals contain their true values, $\beta_a\ge L_a$ and $\beta_b\le U_b$.

If the algorithm separates $a$ above $b$, then:

Combining the inequalities:

Therefore:

So the separated tier boundary is real up to the practical margin. Since the simultaneous intervals hold with probability at least $1-\alpha$, the guarantee holds with probability at least $1-\alpha$. $\square$

This is conservative in the right way. It may keep two truly different models in the same tier if there is not enough evidence. But it will not confidently invent a separation that the intervals do not support.

Update Task Quality

Tasks should also be evaluated. After each comparison, update task noise and discrimination with conservative moving averages:

def ema(old: float, new: float, alpha: float = 0.05) -> float:
    return (1 - alpha) * old + alpha * new


def update_task_stats(
    task_discrimination: dict[str, float],
    task_noise: dict[str, float],
    task_residual_ema: dict[str, float],
    task_variance_ema: dict[str, float],
    task_id: str,
    observed_target: float,
    predicted_target: float,
) -> None:
    residual = abs(observed_target - predicted_target)

    task_residual_ema[task_id] = ema(task_residual_ema[task_id], residual)
    task_variance_ema[task_id] = ema(task_variance_ema[task_id], residual ** 2)
    task_noise[task_id] = sqrt(task_variance_ema[task_id] + 1e-6)

    observed_signal = abs(observed_target - 0.5) * 2
    reliability = 1 / (1 + task_noise[task_id])

    task_discrimination[task_id] = min(
        3.0,
        max(0.25, ema(task_discrimination[task_id], observed_signal * reliability)),
    )

This prevents one dramatic comparison from making a task look permanently magical.

Batch Without Tunnel Vision

When asking for a batch of human grades, do not let one uncertain pair consume the entire batch. Sort candidates by value, then enforce simple diversity limits across models and tasks.

from collections import Counter
from itertools import combinations


def suggest_batch(models: list[str], tasks: list[str], batch_size: int, state: dict):
    candidates = []

    for model_a, model_b in combinations(models, 2):
        for task_id in tasks:
            value = candidate_value(
                beta=state["beta"],
                sigma=state["sigma"],
                task_discrimination=state["task_discrimination"],
                task_noise=state["task_noise"],
                eval_count=state["eval_count"],
                task_eval_count=state["task_eval_count"],
                seen_pair_task=state["seen_pair_task"],
                model_a=model_a,
                model_b=model_b,
                task_id=task_id,
            )
            candidates.append((value, model_a, model_b, task_id))

    candidates.sort(reverse=True)

    batch = []
    used_models = Counter()
    used_tasks = Counter()

    for value, model_a, model_b, task_id in candidates:
        if len(batch) == batch_size:
            break
        if used_models[model_a] >= 4 or used_models[model_b] >= 4:
            continue
        if used_tasks[task_id] >= 4:
            continue

        batch.append((model_a, model_b, task_id, value))
        used_models[model_a] += 1
        used_models[model_b] += 1
        used_tasks[task_id] += 1

    return batch

This keeps the queue useful and varied.

When To Stop

Stop based on decision quality, not a fixed comparison count.

def should_stop(remaining_budget: int, best_candidate_value: float, min_value: float) -> bool:
    if remaining_budget <= 0:
        return True
    if best_candidate_value < min_value:
        return True
    return False

Another practical rule is tier stability:

def stop_when_stable(recent_tiers: list[list[list[str]]]) -> bool:
    if len(recent_tiers) < 3:
        return False
    return recent_tiers[-1] == recent_tiers[-2] == recent_tiers[-3]

If the same tiers survive several batches, more comparisons may not change the product decision.

How Many Comparisons Can This Save?

For 10 models and 20 tasks, brute force is 900 comparisons. An active arena should often land closer to 100 to 300, depending on how close the models are.

The back-of-the-envelope reason is that distinguishing a gap gets quadratically harder as the gap gets smaller.

Suppose two models differ by Bradley-Terry gap:

For a task with discrimination $d_t$, the observable logit gap is:

Near $p=1/2$, each comparison has Fisher information on the order of:

After $k$ effective comparisons, the uncertainty of the gap shrinks like:

To distinguish a practical gap $\delta$, we want:

Solving for $k$:

If we want confidence across many possible model boundaries, add a log factor:

The shape matters more than the constants:

That is why exact rankings are so expensive. Separating models that are 10 Elo apart can require roughly $25\times$ more signal than separating models that are 50 Elo apart:

The biggest savings come from:

• skipping inferred relationships when confidence is already high,
• using every comparison to update global model strength,
• spending more judgments near uncertain tier boundaries,
• downweighting noisy tasks,
• using margin grades instead of binary wins,
• stopping once rankings are stable enough for the decision.

If the top models are almost tied, the system should either spend more comparisons or honestly keep them in the same tier.

The First Version To Ship

The first useful version does not need a full Bayesian optimizer. It needs a tight loop:

• Store pairwise comparison records with -5..+5 grades.
• Update Elo-like scores after each human grade.
• Track per-model uncertainty.
• Track per-task noise and discrimination.
• Suggest high-value model-task pairs.
• Group close models into practical tiers.
• Periodically refit scores from the full match history.

The exhaustive arena asks, "Have we graded every possible comparison?"

The active arena asks, "Would the next comparison change what we believe?"

That is the product difference.