What does this equation mean?

The response, arriving in 2023, was to stop checking against a fixed answer and start recording a preference between two outputs instead. Chatbot Arena built an open platform in which users submit a prompt, receive responses from two anonymized models side by side, and vote for the one they prefer, without knowing which model produced which response; its authors reported collecting more than 240,000 such votes from a broad user base and validated that crowdsourced preferences agreed well with those of a separate panel of expert raters [ 12 ] . Pairwise votes of this kind are naturally converted into a single continuous rating using a Bradley–Terry-style model — the same functional form used for chess Elo ratings — in which the probability that a rater prefers system A over system B is modeled as a logistic function of the difference between two scalar ratings: $P(A \succ B) = \frac{1}{1 + 10^{-(R_A - R_B)/400}}$. That formula encodes a specific, testable assumption: that pairwise preference between any two systems can be explained by each system’s position on one shared scale, and that preferences are therefore transitive — if raters prefer A to B and B to C , they will prefer A to C . This is a real point of methodological disagreement rather than settled fact. Preferences over open-ended text plausibly depend on the task, the rater, and the specific pairing in ways a single scalar rating cannot capture, and a persistently observed intransitive preference cycle would be direct evidence against the model, not just noise around it. The Arena’s own published methodology treats this as an ongoing statistical concern to be monitored, not a solved problem.