A Showdown Ladder Rating Measures the Ladder, Not the Player

A Ladder Rating Looks Like a Skill Score; Showdown’s Own Page Ships It With an Error Bar

Pokémon Showdown’s ladder help page states its rating formulas the way a lab manual states an assay, not the way a game states a high score. A new account starts Elo at 1000. Between 1100 and 1299 the K-factor is 50; at 1300 and above it drops to 40; the rating never falls below a floor of 1000; and above 1400 an inactive account decays daily at 9:00 GMT, losing one point per 100 points above 1500 if it has played one to five games that day and one point per 50 points above 1400 if it has played none at all ^[1]. Running in parallel on the same ladder is a second, independent system: Glicko-1, which starts every account at rating R = 1500 with a rating deviation of RD = 130, constrains RD to the range 25–130, updates on a 24-hour rating period, and carries a published system constant of 6.6775026092 ^[1]. A third figure, GXE — Glicko X-Act Estimate — is defined on the same page in one sentence: “an estimate of your win chance against an average ladder player” ^[1].

Two of those three numbers are a point estimate. The third, RD, is not a skill number at all. It is a confidence measure attached to the skill number, published by the same page, updated by the same algorithm, and routinely ignored by everyone who screenshots a GXE percentage as if it were a grade. This article takes RD exactly as seriously as its publisher’s own documentation says to take it, asks what kind of object a number like that actually is, and then applies to it the one standard the field that invented that kind of object actually uses to decide whether a score can be trusted across different conditions: psychometrics’ construct-validity and measurement-invariance standard. The claim that survives the audit is narrower than “ladder ratings are meaningless” and sharper than “ladder ratings are fine”: Glicko-1 on Showdown is a real, competently specified measurement instrument, and the instrument’s own logic says a rating is trustworthy only relative to the population and metagame it was measured in — which is a problem, because the population and the metagame are exactly the two things a competitive Pokémon ladder never holds still.

Glickman Built the Error Bar to Fix One Specific Complaint About Elo

Glicko was not invented to replace Elo’s arithmetic; it was invented to fix one specific failure mode Elo cannot represent. Mark Glickman states the failure as a worked example on his own explanatory page: two players are both rated 1700, one has just returned to competition after years away and the other plays every weekend, and the first defeats the second. Elo moves both ratings by the same fixed amount regardless of which player’s number was trustworthy going in — “my intuition tells me that (1) the first player’s rating should increase by a large amount… and (2) the second player’s rating should decrease by a small amount” ^[3]. Elo has no mechanism to say that a 1700 is not a real 1700. Glicko-1 adds exactly that mechanism, formalized in Glickman’s 1999 paper in the Journal of the Royal Statistical Society’s Applied Statistics series as a non-iterative approximation to a fully Bayesian dynamic paired-comparison model ^[2].

The mechanism is two coupled numbers, not one. A player’s rating R changes only from game outcomes; the rating deviation RD changes from game outcomes and from the simple passage of time — RD shrinks as a player competes and grows as a player sits idle, because “the more games played, the more information is learned about a player’s ability… as time passes, we become more uncertain about the player’s strength” ^[3]. The update equations, stated with Showdown’s own configured constants substituted in, are

where $d^2$ is a variance term built from the same $g(\mathrm{RD}_j)$ and expected-score factors, $s_j \in \{0, 0.5, 1\}$ is the outcome of game $j$, and between rating periods the deviation itself grows on a fixed schedule, $\mathrm{RD}_{\text{new period}} = \sqrt{\mathrm{RD}_{\text{old}}^2 + c^2}$, clamped to Showdown’s own configured ceiling of 130 rather than Glickman’s generic default ceiling of 350 ^{[2, 3]}. Substitute Showdown’s published $c = 6.6775026092$ and the schedule is fully determined: it is not a metaphorical “uncertainty,” it is a specific number computed the same way every 24 hours for every account on the ladder. A brand-new Showdown account starts at RD = 130 — already at the top of Showdown’s own configured uncertainty range, since that range tops out exactly where the starting value sits — which is the algorithm’s honest way of stating that it knows nothing about a new player yet, before either the player or their opponents log a single result.

The Rating Deviation Is a Standard Error of Measurement, Term for Term

Classical Test Theory gives psychometrics its founding decomposition: an observed score $X$ on any single administration of a test equals a true score $T$ — the score a person would get on average across infinitely many parallel administrations — plus measurement error $E$, so $X = T + E$, with $E$ assumed to average to zero and to be uncorrelated with $T$ across a population of test-takers ^[4]. From that single axiom the whole apparatus of reliability follows: reliability is the proportion of observed-score variance attributable to true-score variance, and the standard error of measurement, $\mathrm{SEM} = \mathrm{SD}_X\sqrt{1 - \text{reliability}}$, is the quantity that turns a single observed score into a defensible interval rather than a point claim — a test-taker’s true score is estimated to lie within roughly $X \pm 1.96 \cdot \mathrm{SEM}$ at 95% confidence ^[4].

Glickman’s RD is this same object, derived from a different starting assumption but converging on an identical use. He states the correspondence in his own words, not as an analogy this article is imposing on his work: a Glicko rating and RD together are “usually more informative to summarize a player’s strength in the form of an interval… report a 95% confidence interval. The lowest value in the interval is the player’s rating minus twice the RD, and the highest value is the player’s rating plus twice the RD” ^[3]. Line up the two constructions and the correspondence is not loose:

This is the article’s original contribution: not the observation that a rating has “uncertainty,” which anyone who has seen a percentage sign after a GXE could guess, but the derivation that Showdown’s published RD update schedule is doing formally, on Glickman’s own stated terms, exactly the job Novick’s 1966 axioms assign to a standard error of measurement — no metaphor, no rounding of two different ideas into looking alike, one construction restated in the other’s notation.

Reliability Is Not Validity, and the Ladder Only Ever Demonstrated the First One

A precise instrument is not automatically a valid one, and psychometrics has kept those two words apart since Lee Cronbach and Paul Meehl’s 1955 paper drew the line: reliability asks whether a measurement is repeatable, while construct validity asks “what psychological constructs account for test performance” at all — whether the number is actually tracking the latent trait its users assume it tracks, checked against a network of other observations the trait ought to predict if it is real ^[5]. A bathroom scale can be extremely reliable, returning the same number to the gram on repeated weighings, while being miscalibrated by five kilograms; reliability bounds the noise, it says nothing about whether the number’s centre is the right centre for the thing you actually wanted to know.

Everything derived above about RD is a reliability result. RD tells a player, correctly, how much noise sits around their current $R$ given how much and how recently they have played. It says nothing about whether $R$ itself is estimating a portable trait called “skill at this game” that would produce the same number if the same player sat down against the same population under different competitive conditions. That second question is a validity question, and Cronbach and Meehl are explicit that answering it requires checking the score against a “nomological network” of related claims it should also satisfy, not just re-measuring it and finding the repeat measurement close to the first one ^[5]. Showdown’s documentation, exhaustive as it is about K-factors and system constants, makes no validity claim at all — it specifies an algorithm, not a psychological construct, and the leap from “this algorithm converges reliably” to “this algorithm measures a stable trait called skill” is a leap Showdown’s own page never asks a reader to make. Competitive players make it anyway, every time a GXE screenshot stands in for an argument about who the better player is.

The distinction bites hardest at exactly the moment a rating gets used for something other than matchmaking. Matchmaking only needs reliability: pair a 1500 against another 1500 and, whatever the number is actually tracking, both players are drawn from roughly the same band of it, so the game is close. Nothing about that use requires the number to mean the same thing next month. Every other use a ladder rating gets put to — settling an argument about who the better player is, seeding an invitational, screenshotting a peak GXE as a credential that outlives the season it was earned in — quietly imports a validity claim that reliability alone cannot support. Cronbach and Meehl’s own examples are drawn from personality and aptitude testing, not games, but the structure of their warning transfers without modification: a score can be reliable enough to bet a ladder placement on and still be the wrong number to bet a comparison across time on, and nothing in Showdown’s documentation, or in ordinary ladder culture, marks that line for the player reading it.

Measurement Invariance Asks Whether the Same Number Means the Same Thing After the Test Changes

The formal tool for checking whether a score means the same thing under different conditions is measurement invariance, and William Meredith’s 1993 paper in Psychometrika gives it a strict hierarchy rather than a yes-or-no label: configural invariance (the same general structure holds), weak or metric invariance (the same items load on the trait with the same weights across groups), strong or scalar invariance (the same weights and the same intercepts, so a raw score is comparable across groups), and strict factorial invariance, which Meredith argues is the level actually “required for fairness/equity to exist” when scores from different groups are compared directly ^[6]. The 2014 Standards for Educational and Psychological Testing, jointly issued by the American Educational Research Association, the American Psychological Association, and the National Council on Measurement in Education, codify the same discipline for practicing test users: a score’s comparability across different populations or testing conditions is a claim that has to be established with evidence, not a default assumption a publisher gets to make simply by keeping the scoring formula fixed ^[7].

A Showdown ladder is a paired-comparison test whose item bank is the current legal metagame — the specific roster of species, movesets, and banned strategies a player’s opponents are drawn from in a given rating period. Meredith’s hierarchy was built for a survey instrument whose items are fixed text; here, the closest analogue to “the items” is the tier’s ruleset, and that ruleset is edited on purpose, by design, whenever a suspect test bans or unbans something. A ban does not merely make the metagame “different” in some vague sense — it removes specific items from the bank the paired-comparison likelihoods are implicitly conditioned on, and Glickman’s model, like every classical paired-comparison model, assumes the population and the comparison structure it is estimating within stays put across a rating period ^[2]. Arthur Berg’s 2020 statistical analysis of chess’s own Elo system works within a domain where that assumption is close to true by construction: chess’s rules have not changed the meaning of a win since long before either rating system existed, so a chess rating’s population-relativity is a comparatively minor concern next to a game whose competitive rule set is redrawn by community vote every few months ^[8]. Showdown does not have that luxury, and nothing in its ladder documentation claims otherwise — the invariance question is simply never asked on the page that publishes the numbers.

Smogon’s Own Suspect Tests Bet That It Does

The strongest evidence against this article’s own reading is institutional, not statistical, and it deserves to be stated at full strength rather than waved off. Smogon runs its tiering process — deciding which species or strategies are banned from a given metagame — through community suspect tests: a vote, open only to players who clear a ladder-standing threshold set fresh for that specific test on that specific metagame’s ladder — Smogon’s own moderators posted, for one representative suspect test, a requirement of 84 GXE reached over at least 42 games on that tier’s current ladder, with a fresh alt account required so the standing could not be borrowed from a previous metagame window ^[10]. That gate is a direct, load-bearing institutional claim that a high ladder rating identifies a player whose in-game judgment is worth weighting more heavily in a governance decision — not merely that the player is currently winning, but that the number reflects something durable enough to trust with a vote that will outlive the rating period it was measured in.

This is a genuine counter-case, not a straw one. If Smogon’s own tiering leadership, who have more first-hand exposure to how ratings behave across metagame shifts than any outside audit, have run this gate for years without abandoning it, that is real evidence the number is doing useful work. But notice what the mechanism actually is: the threshold is never a fixed constant carried over from the last suspect test. It is redrawn, on that metagame’s own ladder, specifically for that test, every time — which is the community’s own tacit admission that a rating from a different metagame window is not a like-for-like substitute for a rating measured inside the window under review. An institution can be right that the number is useful for its purpose — sorting an already-engaged, currently-active voter pool by recent, in-context form — while the same number would fail a stronger claim this article is actually testing: that it estimates a portable skill trait comparable across the metagame boundary a suspect test exists to cross.

The Discriminator: Rank-Correlate the Same Roster Across a Ban

A construct-validity dispute that cannot specify what evidence would settle it is not a scientific dispute, so here is the specific test. Take the cohort of players active on a given tier’s ladder in the week immediately before a major, tier-defining ban takes effect, and record each player’s Glicko-1 rating and RD at that moment. Let the post-ban metagame run long enough for RD to re-settle toward the low end of Showdown’s 25–130 range for players who keep playing regularly — Glickman’s own guidance suggests a rating period works best with “an average of 5-10 games per player” behind it before its output should be trusted ^[3] — then record the same players’ ratings again. Compute the Spearman rank correlation between the two snapshots, using Charles Spearman’s original 1904 formalization of rank-order association as the applicable method for exactly this kind of before/after comparison where the underlying scale’s absolute units are not assumed comparable across conditions ^[9]. A high rank correlation is evidence for the reading Smogon’s governance practice assumes: whatever the rating estimates, it estimates the same relative ordering regardless of which items got banned. A low rank correlation is evidence for this article’s reading: the rating is tracking fit to a specific, temporary item bank, and a ban is precisely the kind of event that should scramble it.

That test is fully specified and, as far as this research pass could establish, not actually runnable from public documentation. Showdown’s own ladder pages return a player’s current standing on request; they do not publish an archived, per-player, per-day rating-history series for the wider ladder population the way a chess federation publishes historical tournament crosstables. Smogon’s monthly usage-statistics releases aggregate which species were used at what rate, which is a different object entirely from an individual’s longitudinal Glicko-1 record. Building the dataset this discriminator needs would require either scraping snapshots from Showdown’s ladder over time before and after a specific announced ban — which this research pass did not do and is not in a position to represent as having done — or obtaining cooperation from Smogon or Showdown’s maintainers directly. The honest position, per this publication’s own rule that an untested discriminator is not to be presented as a completed one, is that the test is designed and falsifiable, and that its data requirement is itself a small finding: an institution making a governance-grade assumption about what its own rating measures has not, as far as this pass could determine, been checked against the one comparison that would confirm or refute it.

What the Number Is Actually Estimating, If Not Cross-Season Skill

None of this is an argument that Glicko-1 on Showdown is broken, miscalibrated, or worth replacing — the algorithm does exactly what Glickman specified, computes exactly what Showdown’s documentation says it computes, and the RD it publishes alongside every rating is a more honest reliability disclosure than most instruments that borrow the word “rating” ever bother to include. The argument is narrower and more specific: a Glicko-1 rating on a game whose competitive item bank is periodically and deliberately rewritten is a measurement of fit to the current metagame with a formally correct error bar, not a validated estimate of a portable skill construct that would travel unchanged across the boundary a ban creates — and the difference between those two claims is exactly the difference Cronbach and Meehl spent their 1955 paper insisting the field keep separate ^[5].

That distinction has a cost outside this one ladder. “Elo” has become a folk unit borrowed across esports, dating apps, and internal engineering leaderboards, usually stripped of the one thing that would make the borrowing responsible: the reliability disclosure Glickman built specifically because Elo’s bare number cannot represent it, and the invariance check that would tell a borrower whether the number survives whatever “ban” their own domain periodically imposes on its own rules. A hiring pipeline that seeds candidates by their competitive-game rating, or a broadcast that reads a peak GXE as a lifetime credential, is making exactly the validity leap this article traces from a specific, well-documented instrument to a claim its own inventor never made for it. The instrument is real. The number it publishes is real. What that number is allowed to mean outside the rating period it was measured in is the question Showdown’s ladder help page never asks, and the one a rank correlation across the next major ban would actually answer.