What does this equation mean?

normalizing each sampled response’s reward against the mean and standard deviation of a group of G responses to the same prompt rather than against a separately trained critic network [ 13 ] . Lambert and colleagues, building the fully open Tulu 3 post-training recipe, named the general approach explicitly, describing it as “a novel method we call Reinforcement Learning with Verifiable Rewards,” and used it alongside supervised fine-tuning and preference optimization rather than as a wholesale replacement for either [ 15 ] . The clearest large-scale demonstration is DeepSeek-R1: Guo and colleagues report training a model with reinforcement learning alone against a purely rule-based reward — combining an answer-correctness check with a format check, and deliberately avoiding a learned reward model because, in their account, a neural reward model “may suffer from reward hacking in large-scale reinforcement learning” — and observed pass@1 accuracy on the AIME 2024 competition-mathematics benchmark rise from 15.6 percent to 71.0 percent over training, reaching 86.7 percent with majority voting across 64 samples [ 14 ] . Formally, the reward itself is as simple as the reward model above was elaborate: $r(x,y) \;=\; \mathbf{1}\!\left[\, y = y^{\star}(x) \,\right]$. The motivation this targets is narrow and specific: it removes the exact failure mode that RLHF’s KL penalty exists to contain. A learned reward model can always be over-optimized past the point where it stops tracking the true objective it was fit to approximate; a verifier that is the objective itself has no such gap to exploit, in principle, so optimization pressure can be applied far more aggressively without the usual proxy-drift risk.