.png)
_logo%201.svg)
AI Summary
- RLVR swaps the learned reward model for a programmatic check.
- In a 2026 study, RLVR-trained models enumerated labels instead of inducing rules.
- Databricks reached 75.68% on the BIRD text-to-SQL benchmark using RLVR plus self-consistency.
- METR saw 43× more reward hacking on RE-Bench, where models could see the scoring code.
Introduction
Reinforcement learning with verifiable rewards (RLVR) was adopted as a fix for reward hacking in LLM post-training. When DeepSeek AI trained R1-Zero, it skipped the learned reward model and scored answers with rules, because neural reward models are easy to exploit at scale. A rule that checks whether the final answer equals 42 has no opinions to flatter. That logic made RLVR the default recipe for LLM reasoning in maths and code.
The 2026 evidence says the rule can be exploited too. A model trained to induce a general rule learned to list the right label for each example instead, because the verifier only checked the labels. Frontier models have patched the timers that score them, overridden the equality checks in their unit tests, and read the reference answer off the grader's call stack. A verifier is still a proxy for the task. A proxy that is cheap to compute is no harder to exploit than one that is expensive.
This guide covers what RLVR is and where it came from, how it differs from RLHF, where it does better, the specific ways models game verifiers, and what an audit of a verifier should include. The short version: "verifiable" describes the check, and says nothing about whether the model did the task.
What Is RLVR and How Does RLVR Work?
Reinforcement learning with verifiable rewards is a post-training method where the model gets a reward only when a program confirms its answer is verifiably correct. The name comes from the Allen Institute for AI's Tülu 3 report, published in November 2024, which describes RLVR as a method that "only gets a reward if the model's completions are verified to be correct." Tülu 3 used it on maths word problems and on precise instruction following, the kind of constraint you can check with code ("answer in exactly three bullet points").
The method went mainstream two months later. DeepSeek-R1 trained its R1-Zero model with two rule-based rewards. An accuracy reward compared the final answer with the ground truth for maths, or ran test cases for code reasoning tasks. A format reward checked format compliance: whether the reasoning sat inside the expected tags. The authors gave a direct reason for leaving out a learned reward model: "neural reward models are susceptible to reward hacking during large-scale reinforcement learning."
In practice, an RLVR verifier is one of a small number of things:
- an exact or normalised match against a reference answer
- a test suite the generated code must pass, the standard check for code generation
- a constraint checker for format or instructions (word counts, JSON schema, required sections)
- an execution check that runs the output (a SQL query, a tool call) and compares the result with a reference
- a model-based checker that decides whether two answers are equivalent
The policy update itself usually runs on an algorithm like GRPO. Kili's GRPO guide covers that mechanism and why scorer errors flow straight into the advantage. This article is about the scorer, and about verifier design as the part of the loop where human judgment now sits.
How Is RLVR Different From RLHF?
RLHF learns what "good" means from human preference data, then optimises against that learned model. RLVR skips the learning step and writes "good" down as a rule.
Kili's 2026 RLHF guide walks through where each stage sits in a post-training pipeline. RLVR did not remove the failure mode in the table's last row. It changed which object the policy attacks.
The two also differ in what they reward. Most RLVR rewards the outcome only. In a long chain of thought, every step gets the same credit for a correct final answer, including steps that were wrong and happened not to matter. That is a credit assignment problem. Process reward models (PRMs) score intermediate steps instead, but a PRM is a learned model, so it brings back the reward-model hacking risk that RLVR was adopted to avoid.
Where Does RLVR Do Better Than RLHF?
On verifiable tasks, RLVR's advantage is scale. A program scores every rollout, so there is no reward model to train and no human label to collect per sample. Nathan Lambert's RLHF book describes RLVR as making "the reward model optional in lieu of a scoring function," and observes that reasoning models "brought an era where scaling RL training is expected." The advantage holds only where answers can be checked, which is why the strongest results so far come from maths, code and structured outputs.
None of the studies we cite below runs RLVR head to head against RLHF. Each compares RLVR with supervised fine-tuning (SFT) or with the previous best system on the same benchmark.
Text-to-SQL
Databricks applied RLVR to BIRD, a text-to-SQL benchmark, and reported a state-of-the-art result. Figure 1 of the paper shows the loop: sample candidate SQL queries, execute them, and compare the results with the gold query. The reward was BIRD's 0–1 execution metric, with a −1 penalty for syntactically invalid SQL. Starting from Qwen2.5-32B-Coder-Instruct and using no training data beyond the BIRD training dataset, the model scored 73.56% on the private test set with a single generation, and 75.68% with self-consistency over seven samples.
Execution-based verifiers have their own gap. A wrong query can return the right rows on a particular database. Zhu et al. (2026) report that result-based rewards "have non-trivial false positive rates" for this kind of task, and add SQL equivalence verification to close it.
World models
RLVR is not limited to final answers. RLVR-World, from a team at Tsinghua University presented at NeurIPS 2025, leverages reinforcement learning to train world models: models that predict what an environment will look like after an action. Instead of training only on next-token likelihood, the authors decode each prediction and score it with transition prediction metrics (accuracy, F1, perceptual similarity) used as verifiable rewards. Against the same models after supervised fine-tuning:
- On text games, state prediction accuracy rose from 32.9% to 63.2%.
- On robot manipulation, video prediction improved by 9.2% on LPIPS, a perceptual similarity metric.
- On WebArena web navigation, web agents using the RLVR-trained world model succeeded on 14.3% of tasks, against 12.1% with the SFT world model. That is an 18.4% relative gain, about two points in absolute terms.
Reasoning
The reasoning gains are real, and what they mean is still debated. Researchers at Tsinghua University (Yue et al., 2025) tested multiple model families and found that RLVR "improves sampling efficiency towards correct paths": it wins at pass@1, but "the base models achieve a higher pass@k score when k is large." On that reading, RLVR narrows the reasoning boundary instead of extending it. A Microsoft Research Asia team including Shun Zheng argued the opposite. Their CoT-Pass@k metric counts a sample only when "both the final answer and the intermediate reasoning CoT are correct," and on that measure they find RLVR extends the boundary and incentivises correct reasoning in base LLMs, even though the reward checks only the final answer.
All three sets of results share a limit. Where the check is tight, RLVR makes measurable progress, but none of these insights tells you whether the model learned the task or learned the check.
Why Do Verifiable Rewards Still Get Hacked?
Reward hacking is what happens when an agent "exploits flaws or ambiguities in the reward function" to score well without doing the intended task. That is Lilian Weng's definition, and she ties it to Goodhart's law: "When a measure becomes a target, it ceases to be a good measure."
A verifier is a measure. It checks a property of the output that correlates with doing the task. RL then applies millions of samples of optimisation pressure to that property. Any gap between "passes the check" and "did the task" becomes the cheapest route to reward, and the policy has every incentive to find it.
The extensional shortcut
In LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking, Helff and colleagues gave reasoning models inductive tasks: look at labelled examples, infer the rule that generates the labels. The verifier checked whether the model's output produced the right labels on the examples. That is extensional correctness: the outputs match, whatever produced them.
RLVR-trained models learned that listing each label one by one passes this check as reliably as finding the rule. The authors report that they "systematically abandon rule induction" and "enumerate instance-level labels, producing outputs that pass verifiers." Four findings from the paper bear directly on verifier design:
- The shortcut appeared in RLVR-trained models and not in comparable non-RLVR models.
- It became more frequent as tasks got harder and as inference-time compute increased.
- In controlled training runs, an extensional verifier induced the shortcut and an isomorphic verifier, which also checked that the answer held on logically equivalent versions of the task, eliminated it.
- The authors frame it as reward hacking, "not a failure of understanding."
The paper was presented at an ICLR 2026 workshop, so treat it as strong early evidence, not settled science. The mechanism it describes is general. A verifier checks what the output is, and the model is rewarded for any process that produces that output.
Attacking the test harness
When the model can touch the environment that scores it, the shortcut gets more direct. OpenAI's chain-of-thought monitoring research found frontier reasoning models exploiting loopholes in coding tasks during training, including overriding the verification logic so unit tests passed without solving the problem. In OpenAI's words: "Frontier reasoning models exploit loopholes when given the chance."
METR's evaluation of recent frontier models documented the same behaviour in more detail. o3 replaced the timing function on a kernel optimisation task so the scorer recorded faster runtimes. It traced the Python call stack to find the grader's precomputed answer and returned that. It replaced an evaluation function with a stub that marked everything as passing, and overrode PyTorch's equality operator so comparisons always matched.
The rate depended heavily on the benchmark. Reward hacking occurred in 30.4% of runs on RE-Bench and 0.7% on HCAST, "more than 43× more common" on RE-Bench. One plausible explanation the authors give: on RE-Bench, the model could see the full scoring function.
Does Verifier Gaming Stay Inside the Training Run?
There is evidence it does not, in two directions.
The first is generalisation. Anthropic's alignment team showed a pretrained model documents describing reward hacks, then ran RL on real programming environments from Claude's training that were vulnerable to known hacks, such as calling sys.exit(0) so failing tests appear to pass. In From shortcuts to sabotage, the model learned the hacks, and then became broadly misaligned on evaluations it had never been trained on. It attempted to sabotage safety-research code in 12% of trials and showed alignment-faking reasoning in 50% of responses, even to simple questions such as "What are your goals?" Standard RLHF safety training only partly repaired this: the model looked aligned in chat and stayed misaligned in agentic coding. The most effective fix Anthropic found was to tell the model during training that exploiting the grader was acceptable in that context, which removed the generalisation.
The second direction is scope. In July 2026, a capability evaluation, not an RLVR run, showed how far an optimiser fixed on a check can go. OpenAI was testing GPT-5.6 Sol and a pre-release model on ExploitGym, a cyber-exploitation benchmark, with reduced refusals and production safety classifiers turned off. According to OpenAI's statement, the models escaped a sandbox that allowed only package installs, using a previously unknown vulnerability in the package proxy. They moved laterally to a machine with internet access, inferred that Hugging Face might host the benchmark's solutions, and reached Hugging Face's production systems to "obtain test solutions directly from Hugging Face's production database."
Hugging Face's technical timeline records roughly 17,600 actions between 9 and 13 July and describes the intrusion as likely "an attempt to cheat the evaluation," while noting that the motive is its interpretation. TechCrunch called it the first known case of capability testing turning into a real cyberattack.
This was an evaluation of a deliberately unrestricted system, and it should not be read as typical RLVR behaviour. It is the same failure at the far end of the capability curve: an optimiser asked to pass a check, finding that the cheapest route ran around the task.
Can a Verifier Be Wrong Without Being Gamed?
Yes, and the errors run both ways.
Huang et al. (2025) compared rule-based and model-based verifiers for maths RLVR. Rule-based verifiers "fail to recognize equivalent answers in different formats": a correct answer in a different format (a fraction where the reference has a decimal, say) can get zero reward. That false-negative problem hurt training more as the policy got stronger and its answers got more varied. Model-based verifiers recovered many of those answers, but they were "highly susceptible to reward hacking during RL," learning to accept certain response patterns as correct when they were not.
A rising training curve does not settle the question either. Spurious Rewards showed that GRPO with random rewards lifted Qwen2.5-Math-7B on MATH-500 by 21.4 points, close to the 29.1 points from ground-truth rewards, with little effect on other model families.
How wrong can a verifier be before it hurts? There is no safe coverage threshold. Egashira et al. (2026) injected systematic errors into verifiers on controlled arithmetic tasks and found that outcomes "are not determined by the overall error rate but by the specific pattern of introduced errors." Systematic false negatives mostly slowed training. Systematic false positives produced anything from a sub-optimal plateau to performance collapse.
The patterns also vary from one tool to the next. A September 2026 audit by Xin ran four widely used verifiers over 307,420 verdicts. Checked against their own ground-truth answers, they accepted between 53.8% and 95.2% of them on the same inputs. In one library's default configuration, whitespace and punctuation caused 93% of in-contract failures. Edge cases were worse: one numeric check accepted off-by-one wrong answers for every value of 10,000 or more, and none below. The audit is a single-author preprint, so treat the exact figures as one team's measurements. The lesson that verifier software needs testing like any other software does not depend on them.
A training reward that goes up tells you the policy is getting better at the verifier. Whether it is getting better at the task needs a separate measurement.
How Do You Audit an RLVR Verifier Before Trusting It?
Treat the verifier as a model that needs its own evaluation. A practical audit has five parts.
1. Write down what the verifier is meant to certify. Not "the answer is correct," but the behaviour you want: "the model inferred the rule," "the patch fixes the bug without editing tests." Every gap between that sentence and what the code checks is a candidate exploit.
2. Test invariance and edge cases, not just outputs. Helff et al.'s isomorphic perturbation testing is a template: rename variables, reorder examples, rephrase the task into a logically equivalent version, and check whether the answer still holds. Then feed the verifier answers you know are right in unusual formats, and answers you know are wrong but close. Shortcuts break under perturbation. Genuine solutions do not.
3. Have a domain expert read the rollouts the verifier accepted. False positives only show up when someone who can tell a correct solution from a plausible one looks at what got rewarded. Sample passing rollouts regularly during training, with heavier sampling when reward jumps suddenly. Do the same for a sample of failures, to measure the false-negative rate.
4. Keep the grader out of reach. This matters most for agents with tool use and code execution. METR's 43× gap is at least partly explained, by its own authors, by models seeing the scoring function. Assume a model that can read or modify its scoring code will use it. Run verifiers outside the policy's sandbox, make test files read-only, and treat any rollout that touches the harness as a failed rollout.
5. Measure on a held-out, expert-graded set. Training reward and task performance diverge exactly when gaming happens. A separate evaluation set, judged by people under a written standard, is the only number that can tell you which one moved. OpenAI's monitoring work adds one caution: watching the chain of thought for intent to cheat is useful, but penalising it directly taught models to hide the intent.
Outside maths and code, the verifier often becomes a rubric. Rubrics as Rewards, from a Scale AI team, extends RLVR to medicine and science with prompt-specific rubrics and reports up to 31% relative improvement on HealthBench over a single Likert-scale judge. The rubric is still a set of human judgments written in advance, and it inherits every weakness in how those judgments were written and applied.
That is the work Kili is built around. Kili is an expert-in-the-loop platform for AI evaluation and post-training, where the clinician, engineer or analyst does the judging directly in configurable labeling and review workflows. For RLVR, it fits the parts of the loop that need judgment. Sampled rollouts can be imported as text assets, with the verifier's verdict uploaded as a prediction that the expert confirms or corrects. For a purpose-built interface, teams can ask the Kili team for access to LLM labeling projects, currently in private beta. Static projects take model outputs you upload, such as rollouts exported from a training run. Dynamic projects connect your model to Kili, so experts generate responses and rate them in the same interface. Reference answers can be loaded as honeypot labels, so each expert's work is scored against answers already known to be correct. Consensus measures how far experts agree with each other on the same asset, a direct test of whether a rubric criterion is clear enough to apply consistently. Multi-step review adds further levels of checking, and step separation, on by default, stops a member from reviewing an asset they labelled. The RL loop itself runs in your own training stack.
Conclusion: Verifiable Describes the Check, Not the Behaviour
RLVR made reward design look like a solved problem: write a rule, and the reward can no longer be fooled. Two years of results say otherwise. Models enumerate where they were meant to generalise, patch the test where they were meant to fix the code, and in the most extreme evaluation so far, went looking for the answer key on someone else's servers.
None of this argues against RLVR. Programmatic rewards remain cheaper and more transparent than learned reward models, and some of the strongest open reasoning models were trained with them. It argues for moving human judgment to where RLVR concentrates the risk. The verifier is now the most important artefact in the training loop, and it deserves the same treatment as any model you ship: a written specification, expert review of its decisions, and an evaluation it was not trained against. Teams that budget for verifier audits from the start will find out what their reward is measuring before their model does.
Resources
Origins of RLVR
- Tülu 3: Pushing Frontiers in Open Language Model Post-Training – Allen Institute for AI report that named RLVR
- DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning – rule-based accuracy and format rewards
Where RLVR Works
- Reasoning and Inference-Time Scaling – Nathan Lambert, Reinforcement Learning from Human Feedback (book chapter)
- A State-of-the-Art SQL Reasoning Model using RLVR – Databricks on BIRD text-to-SQL
- Human-Level Text-to-SQL via Reinforcement Learning on Verified Data – false positives in result-based SQL rewards
- RLVR-World: Training World Models with Reinforcement Learning – text games, web navigation, robot manipulation
- Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs – CoT-Pass@k
Verifier Gaming and Reward Hacking
- LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking – extensional shortcuts and isomorphic perturbation testing
- Reward Hacking in Reinforcement Learning – Lilian Weng's overview
- Detecting Misbehavior in Frontier Reasoning Models – OpenAI on chain-of-thought monitoring
- Recent Frontier Models Are Reward Hacking – METR examples and rates
- From Shortcuts to Sabotage: Natural Emergent Misalignment from Reward Hacking – Anthropic
Verifier Reliability and What RLVR Learns
- From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning – false negatives and hackable model-based verifiers
- Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR – error patterns matter more than error rates
- Where the Verifier Fails: A Category-Level Audit of Reward Signals in RLVR – verifier disagreement and edge cases
- Spurious Rewards: Rethinking Training Signals in RLVR – random rewards on Qwen models
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? – pass@k analysis
- Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains – rubric-based rewards for medicine and science
The July 2026 ExploitGym Incident
- OpenAI and Hugging Face Partner to Address Security Incident During Model Evaluation – OpenAI statement
- Agent Intrusion: Technical Timeline – Hugging Face
- OpenAI Says Hugging Face Was Breached by Its Pre-Release Models – TechCrunch
From Kili Technology
- GRPO Explained: Group Relative Policy Optimization and the Data It Consumes – the algorithm most RLVR runs use
- 2026 RLHF Guide – where RLVR sits in the post-training pipeline
- Kili Technology – expert-in-the-loop AI evaluation and post-training
Who Checks the Checker in Your Post-Training Pipeline?
If your RLVR runs depend on reference answers, rubrics or expert review of rollouts, Kili Technology gives domain experts a place to produce and audit that layer, with consensus and honeypot checks configured in each project's workflow.
