Symbol s

The passage around this formula

Here $\theta$ denotes the model’s parameters and H denotes everything about the evaluation procedure that is not the model itself — the exact prompt template, the number of few-shot examples, whether a chat template is applied, how the model’s free-text output is parsed into a scored answer, and the decoding configuration. A vendor publishes one value of s . An independent lab computing its own s' is implicitly testing whether f($\theta$, H) = f($\theta$', H') , and a mismatch could in principle come from either argument.

Read this part in the article →