Does Getting the Right Answer Improve Reasoning? The Promise and Limits of RLVR

Related posts — Reflections on DeepSeek-R1, PPO, and GRPO · DPO · How LLMs Are Trained

Recent reasoning models can behave strangely when asked to translate. The same model that will patiently revise a mathematical solution, try another implementation until the tests pass, or recover from several failures in an agent environment may make an already natural translation awkward by revising it once too often. Sometimes it adds an explanation that was never in the source.

I first treated this as a matter of model quality. Yet scores on math and coding benchmarks kept rising, and agents kept learning to handle longer tasks, while it was less obvious that the most basic language abilities were improving at the same pace. Perhaps the confusion came from treating capability as a single score.

That question grew sharper as I read about Reinforcement Learning with Verifiable Rewards, or RLVR. RLVR automatically checks a model's answer and rewards responses that succeed. It is compelling precisely because long reasoning and self-correction can emerge without a person writing out every successful trajectory.

But is arriving at the right answer the same as reasoning correctly? If a verifier sees only the answer at the end, how does it distinguish a sound derivation from a lucky one, or a direct solution from a wasteful detour? And does training on verifiable math and code help with open-ended language tasks such as translation and writing?

My question, then, is where RLVR's boundary lies. Without that boundary, it is too easy to translate higher benchmark scores into a much broader claim about reasoning.

RLVR Is Not an Algorithm

The name RLVR came into wide use with Tülu 3, and DeepSeek-R1 soon made large-scale reasoning RL much more visible. The two projects use the same general kind of reward, but they do not update the policy in the same way.

Tülu 3 used PPO. DeepSeek-R1-Zero used GRPO. RLVR describes where the reward comes from; PPO and GRPO describe how that reward updates the policy. Treating RLVR and GRPO as interchangeable hides an important distinction between the training signal and the policy-update algorithm.

A minimal verifier can be written as

$$r(x,y)=v(x,y) = \begin{cases} 1, & \text{if response }y\text{ passes the verifier for input }x \\ 0, & \text{otherwise.} \end{cases}$$

For a math problem, the verifier may compare the final answer or test symbolic equivalence. For code, it can compile the program and run unit tests. An instruction-following task may check a JSON schema, a word limit, or the presence of required fields. Programs can also verify SQL execution results, formal proofs, game scores, and the state reached by an agent.

The reward need not be binary. Passing seven of ten tests can produce a score of 0.7, and a constraint problem can award the fraction of conditions satisfied. Exact string matching is only one implementation. What matters is whether the reward can be computed consistently without asking a person to read every response.

Tülu 3 verified answers on GSM8K and MATH and constraints on IFEval. Adding RLVR after SFT and DPO improved the targeted math and instruction-following skills, with some gains on tasks that were not part of the RL objective. The average across all evaluations did not rise in lockstep. From the beginning, RLVR looked less like a replacement for the whole post-training stack and more like a final stage for pushing specific, verifiable capabilities.

What Signals SFT, DPO, and RLVR Learn From

The same response carries different training information under each method.

Even if people label every preference pair used by DPO, the resulting preference is not automatically universal. People may largely agree about factual errors or clear harm. They disagree more often about style, answer length, humor, when a refusal is appropriate, and questions involving competing values.

Diverging Preferences found that annotator disagreement in preference data was not merely labeling noise. Genuine differences appeared around underspecified tasks, response style, and refusals, yet common reward models and LLM judges tended to collapse them into a single majority choice.

When judgments from several annotators are collapsed into one chosen label, a response preferred by 51 percent can be recorded in the same way as one selected unanimously. The target is therefore closer to the empirical preference observed under a particular annotator pool, rubric, and candidate distribution than to a universal human taste. Preference learning remains useful for subjective tasks, but satisfaction has a ceiling when information about the user and context is discarded and many preferences are compressed into one average reward.

RLVR replaces annotator-level preference disagreement with a programmatic verifier. The decision becomes reproducible, but the choice of specification—what the verifier checks—remains a design decision. Verifier coverage is also narrower. DPO and RLVR are not simply competing solutions to the same problem; they incur different costs in human judgment and verifier coverage.

What DeepSeek-R1-Zero Actually Rewarded

DeepSeek-R1-Zero applied GRPO to DeepSeek-V3-Base without a cold-start SFT stage. For reasoning data, it used rule-based signals instead of a neural reward model. The paper describes two main terms: accuracy and format.

$$R_{\text{rule}} = R_{\text{accuracy}} + R_{\text{format}}.$$

Math answers were checked in a specified final-answer format, while code was evaluated with a compiler and test cases. The format reward checked whether the reasoning was enclosed in the designated <think>...</think> tags. R1-Zero did not ask a human or a process reward model to grade every step of the derivation.

GRPO samples several responses to the same prompt and reinforces those with higher rewards relative to the group. In simplified form, the advantage for response $i$ is

$$A_i = \frac{ r_i-\operatorname{mean}(r_1,\ldots,r_G) }{ \operatorname{std}(r_1,\ldots,r_G)+\epsilon }.$$

The group mean acts as a baseline, removing the need for a separate critic model. A prompt is informative when some sampled answers succeed and others fail. If every answer is wrong, or every answer is correct, the relative signal from that group becomes weak. This is one reason RLVR datasets often need problems within reach of the current policy: the policy has to succeed at least occasionally before success can be reinforced.

The trajectory-level advantage derived from the outcome is then used in the token log-probability updates for that response. Without a separate process signal, the update cannot identify which step earned the outcome and which step merely survived alongside it.

Even this sparse signal produced patterns of revision, backtracking, checking, and longer reasoning in R1-Zero. There is no need to interpret them as human-like intention or self-awareness. A simpler account is that the policy shifted toward token trajectories that had repeatedly appeared with higher rewards.

The full DeepSeek-R1 recipe did not rely on RLVR alone. It added cold-start SFT, reasoning RL, another SFT stage with roughly 600,000 reasoning samples and 200,000 non-reasoning samples, and a final RL stage that included general helpfulness and harmlessness. These stages were introduced to mitigate readability, language mixing, and general-task weaknesses and improved general benchmarks. The paper still lists language mixing outside Chinese and English as a limitation.

The Gap Between a Correct Answer and Correct Reasoning

A correct final answer leaves at least four questions unanswered.

Outcome Validity

Did the verifier identify the answer correctly? A brittle parser may treat $\frac{12}{36}$ and $\frac{1}{3}$ as different. An incomplete test suite can accept a broken program. A verifiable reward is never stronger than the specification implemented by its verifier.

Validity of the Written Process

Does each statement follow from what came before? A derivation can make the same sign error twice and land on the correct result. A response can attach an unsupported explanation to a correct answer. Outcome reward does not distinguish these false positives from a nearly correct solution that makes one arithmetic mistake at the end.

Causal Faithfulness

The chain of thought shown in text may not be what actually determined the answer. If changing or removing parts of the written reasoning leaves the answer unchanged, a readable explanation should not automatically be treated as a transcript of the model's computation. In experiments on the Qwen2.5 series and ReasoningGym tasks, the 2026 preprint Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning reports that higher task accuracy after RLVR did not reliably bring higher causal importance of reasoning tokens or make the written reasoning sufficient to derive the answer. The result still needs replication across more models and tasks, but it separates a distinction that outcome accuracy alone cannot measure.

Optimality

There is rarely one best valid solution. The shortest path, the path least likely to contain an error, the path easiest for a person to audit, and the path with the lowest compute cost may all differ. A strong length penalty can produce terse leaps. A reward for checking can attach a long monologue to an easy problem. Once we call a trajectory optimal, preference enters again through the choice of what to preserve and what cost to minimize.

Does Process Reward Solve the Problem?

The most direct response is process supervision: evaluate each intermediate step. Let's Verify Step by Step collected human labels on steps in mathematical solutions and trained a process reward model. In best-of-N selection, the PRM chose good solutions more reliably than a reward model trained only on final outcomes.

That experiment used the PRM mainly to rank generated solutions, rather than demonstrating direct RL of the generator with PRM rewards. Human or LLM judgments can provide finer feedback over natural-language steps, but the signal is then a learned assessment rather than a verifiable reward in the strict sense.

Intermediate rewards become more compelling when the states themselves can be checked. A proof assistant can validate each formal inference. An agent environment can inspect the state after a tool call. Verifiable Process Rewards instantiates this idea with search-, constraint-, and posterior-based checks, separating useful steps inside failed trajectories from bad steps inside trajectories that happened to succeed.

The following is not the objective proposed in the VPR paper. It is a generic example of combining outcome, step, and cost signals:

$$R(\tau) = R_{\text{outcome}} + \alpha\sum_t R_{\text{step},t} - \beta C(\tau),$$

where $R_{\text{outcome}}$ measures final success, $R_{\text{step},t}$ measures validity or progress at step $t$, and $C(\tau)$ represents token or execution cost. Every added term introduces another source of measurement error and another weight to choose. In VPR's Tic-Tac-Toe oracle-quality ablation, an intermediate verifier with a small search budget made performance worse than the base model. A dense reward is not automatically an accurate reward.

The current landscape fits into two rough axes.

The upper left is cheap and precise but coarse in credit assignment. The lower right carries richer information while increasing judgment cost and the opportunities for reward hacking. A practical system can mix these signals according to the cost of failure and the reliability of each verifier instead of forcing every task into one cell.

Does RLVR Create New Capabilities?

R1-Zero can give the impression that RL discovered reasoning strategies absent from the base model. It is difficult, however, to separate a genuinely new path from a path the base model already produced rarely and RL made common.

The literature has not settled this question. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? observed that RLVR models improved at small $k$, including Pass@1, while base models could retain higher Pass@k at sufficiently large $k$. The authors argue that current RLVR often reweights an existing distribution instead of expanding its support. Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs counters that final-answer Pass@K misses the quality of the path. Using CoT-Pass@K, which counts both a correct result and a correct intermediate derivation, it finds evidence that RLVR can extend the reasoning boundary. That process evaluation also relies on model judgment, so the definition of success remains part of the result.

An even stranger finding comes from Spurious Rewards. On Qwen2.5-Math-7B, random rewards improved MATH-500 by 21.4 percentage points, not far from the 29.1-point gain with ground-truth rewards. The effect did not reproduce on Llama 3 or OLMo 2. One interpretation is that the optimization amplified code-reasoning behavior already common and useful in that particular Qwen model.

The point is that RLVR may play different roles across model families and pre-training regimes. In some models, eliciting and self-distilling a latent behavior may explain much of the gain. How much an accurate reward matters has to be read alongside the base model's prior distribution.

Does Better Math Mean Better Translation?

If capability is one scalar, we may expect an improvement in math to raise language ability with it. A model is better represented by a vector of skills.

$$\mathbf{c} = (c_{\text{math}},c_{\text{code}},c_{\text{agent}},c_{\text{translation}},c_{\text{writing}},\ldots).$$

One training objective can move several coordinates together. Tülu 3's RLVR stage produced small point-estimate gains on BigBenchHard, DROP, and AlpacaEval 2 despite not directly optimizing those evaluations. The paper did not report significance tests, so these numbers are compatible with transfer but do not establish it. Behaviors such as decomposing a problem, checking an answer, and following constraints may transfer across tasks.

Transfer is not automatic. Repeated optimization on math and coding prompts may shift the output distribution toward a narrow training regime and interfere with behavior used on general language data. That mechanism alone does not establish that RLVR damaged language ability. The question needs controlled evaluation across models and domains.

There are at least two explanations for the translation failures I notice in reasoning models. Post-training may have damaged multilingual representations or fluency. Alternatively, the language ability may remain intact while an inference policy built around long reasoning and repeated revision interferes with a task that often benefits from direct generation.

Test-Time Scaling of Reasoning Models for Machine Translation evaluated 12 reasoning models across several translation benchmarks. More reasoning brought limited and inconsistent gains in direct translation. Forcing models to continue past their natural stopping point consistently hurt quality. DRT models fine-tuned on MetaphorTrans improved consistently on that in-domain task up to their self-determined stopping depth. The relationship disappeared on out-of-domain tasks such as WMT24-Literary. Reasoning also worked reliably in post-editing, where the model first received a draft and then corrected it.

Translation still contains problems that benefit from reasoning: ambiguous references, long context, specialist terms, and cultural expressions. A free-form long chain of thought borrowed from math may be less useful than a translation-specific procedure: draft once, check adequacy, repair fluency, and stop when further edits no longer help. When and what the model checks matters more than simply thinking for longer.

DeepSeek-R1's later stages point in the same direction. After reasoning RL, the model was trained again on non-reasoning data that included translation and writing, followed by a final stage with general preference rewards. A production generalist model uses a mixture of SFT, preference learning, verifiable rewards, and broad evaluation. RLVR alone is not the recipe.

How Far Should We Trust RLVR?

RLVR has a clear home: tasks whose success condition can be written down, where the model can attempt several solutions and a verifier can return a consistent signal. The category extends beyond math and code to SQL, formal proofs, constrained planning, tool use, and agents operating in simulators.

Open-ended writing, advice, and value judgments are different. We can manufacture a reward by choosing one reference answer or attaching an LLM judge, but doing so weakens the property that made the reward verifiable. Once model judgment is needed to extend verifier coverage, the boundary between RLVR and RLHF or RLAIF becomes less clean.

To me, RLVR's strongest result is that where accurate feedback can be produced cheaply, a policy can explore at scale without humans writing every successful trajectory. It does not establish that the LLM has become generally goal-directed.

Its success remains tied to what can be checked. An outcome verifier can reinforce behavior that reaches correct answers, but validity, faithfulness, and efficiency remain separate. Process rewards reveal more of the path, then leave us with the job of validating the process verifier. The hard problem has moved into the design of the checker.

Rewarding correct answers can make a model produce correct answers more often. That alone is not enough to say that it thinks in a better way. Before treating RLVR as evidence of general intelligence, we should separate what the reward observed from what it never saw.

References

• Tülu 3: Pushing Frontiers in Open Language Model Post-Training
• DeepSeek-R1 Incentivizes Reasoning in LLMs through Reinforcement Learning
• Let's Verify Step by Step
• Diverging Preferences: When do Annotators Disagree and do Models Know?
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
• Spurious Rewards: Rethinking Training Signals in RLVR
• Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning
• Verifiable Process Rewards for Agentic Reasoning
• Test-Time Scaling of Reasoning Models for Machine Translation