Methodology

Last verified/updated: 2026-10-06

AI Video Arena is a blind pairwise preference leaderboard for AI video models. No votes have been collected yet, so nothing on this site is a rating — the pages currently publish a vendor-sourced catalogue and nothing more. This page documents, in advance, every step from a future vote to a published number, in enough detail that the whole pipeline can be re-implemented from scratch and checked against what we eventually ship.

Why the method is published before the data

Publishing the procedure first is the only protection against fitting the method to the result. Everything below is fixed before a single vote exists: the model family, the interval procedure, the minimum-vote threshold and the rule for what gets shown when there is not enough data. If any of it changes later, the change goes on the corrections page with a date, not into a silent edit.

Until voting starts, the leaderboard pages show a catalogue and say plainly that there are no votes. There is no provisional score, no seeded rating and no editorial ranking standing in for one.

From votes to scores: Bradley-Terry with an Elo-style scale

A vote is a partial ordering of two items. Aggregate enough of them and you can fit each model a strength parameter; the standard choice for pairwise data is the Bradley-Terry model, which estimates strengths by maximum likelihood and is the statistical engine behind the better-known chatbot arenas (the general approach follows the public pairwise leaderboard at arena.ai (LMArena, formerly lmarena.ai), adapted here to video).

We will report strengths on an Elo-style scale rather than raw probabilities because the scale is intuitive: a 40-point gap predicts roughly a 56% win rate — our own calculation from the standard Elo expected-score formula, 1/(1+10^-40/400) ≈ 0.56, checked 2026-09-29 — and doubling gaps compounds on the same scale readers know from chess. One honest caveat on comparability: every snapshot will re-centre the reported scale on that snapshot's pool mean (1000), so within a snapshot the numbers will be directly comparable, but cross-snapshot movement will reflect the pool's composition as well as any model's change — read long-term drift as relative to the field, not as an absolute constant.

Ties and 'both bad' votes will count as half-wins for each side; skips are discarded entirely. A model that is frequently skipped will not be penalised, but its interval will grow no faster either.

Uncertainty: bootstrap 95% confidence intervals

A point estimate without an uncertainty band invites over-reading, so every score this site ever publishes will ship with a 95% bootstrap confidence interval. The procedure: resample the battle pool with replacement, refit the model on each resample, and take the 2.5th and 97.5th percentiles of the resulting score distribution. Intervals widen approximately with 1/√votes, which is why vote volume is the leaderboard's real currency.

Two consequences follow. First, models whose intervals overlap are not statistically separated, whatever their nominal ranks say — the interval column exists to make that overlap visible at a glance. Second, intervals will be computed per category, because a model with ten thousand text-to-video votes and forty video-edit votes has very different certainty in the two tabs.

Rank spread: the honest rank column

When ranks appear, the rank column will sort models by point estimate, but point-estimate ranks overstate certainty when intervals overlap. The rank spread column will state, for each model, the range of ranks its own interval is consistent with: the best possible rank is one plus the number of models whose entire interval sits above its upper bound, and the worst possible rank is the number of models whose intervals reach its lower bound.

A model showing '3 – 5' would be telling you the data has placed it in a three-way argument. We considered showing only this column and dropping point ranks entirely; we kept both because readers navigate by ranks but should decide by spreads.

Minimum-vote threshold and 'not enough votes yet'

No model will appear on a public leaderboard below a minimum-vote threshold (300 votes in the current configuration). Below the threshold a model will render as 'not enough votes yet' rather than with a score, because early votes are the noisiest — a model rated by fifteen fans or fifteen detractors can land anywhere.

The build enforces this mechanically. The static generator currently refuses to run in anything other than pre-launch mode: if a score, a confidence interval, a vote count or a win share ever appears in the site's data files while the arena is marked pre-launch, the build fails instead of publishing it. When a real snapshot exists, the same guard will require every listed model to clear the threshold in every published category pool, so a half-seeded database can never leak into search results. The threshold is a property of the data file, not of hand-edited pages, and it will be raised as total volume grows.

Vote quality and anti-abuse

A leaderboard is only as honest as its worst ballot, so several filters will run before a vote enters the fit. Votes are deduplicated per voter per battle using a salted daily hash of IP and user agent — the raw IP is never stored (see the privacy page). Rate limiting caps votes per voter per window, and a change-sensitive signature binds each vote to the exact battle pair it answers, so ballots can't be replayed against other matchups.

A human-verification challenge sits in front of the vote endpoint. Two further safeguards are designed and documented but not yet enabled in the rating job, and we say so rather than imply otherwise: dropping battles with anomalous skip rates (skips are discarded before storage, so the filter has no data to act on — storing skip telemetry first is on the backend roadmap) and flagging — rather than silently deleting — suspicious vote clusters for human review. What is already implemented is the list above: dedupe, rate limits and signed battle tokens. We publish the snapshot behind every page so anomalies are inspectable by anyone, including the vendors being listed.

External crowd rankings (Design Arena)

Besides this arena's own (future) figures, the site quotes one external crowd ranking: the public video leaderboard run by Design Arena (checked 2026-10-21). Their method is close to the one documented above: Elo ratings start at 1200 and are calculated with the standard Bradley-Terry model, the published margin is +/- one standard error from that fit, and each model's vote count is the wins and losses it has accumulated in blind pairwise preference battles on their site.

Every one of those figures is Design Arena's, not ours. Where they appear — the crowd-votes table on the home page and the crowd-stats cards on model pages — they are quoted from their board with a link and the date we checked it, and the markup marks them as external data so our own pre-launch guardrails keep applying to everything else on the page. The two boards are independent: their numbers are not an input to our fit, and our arena publishes no ratings of its own until its voting starts.

One reading note that applies to any crowd board, theirs included: a board entry is evidence that a model exists on that platform, not that it is generally available, and their roster includes variants and legacy models this catalogue does not cover. Where their board lists one of our models we link the two; where it does not, the model page says so.