CSCI 8945 · Post training
Aligning LLMs: PPO vs DPO vs GRPO
High-level math, intuition, and term-by-term breakdowns of the three policy-optimization methods behind modern LLM alignment: Proximal Policy Optimization, Direct Preference Optimization, and Group Relative Policy Optimization.
Background: Key RL Concepts
Before diving into the three methods, here is a quick refresher on the reinforcement learning vocabulary they share. If you are coming from supervised learning, the core shift is this: instead of learning from fixed (input, label) pairs, an agent learns by taking actions in an environment and observing rewards.
| State s | A snapshot of the world at a given moment. In the LLM setting, the state at time step t is the prompt x concatenated with all tokens generated so far: st = (x, y1, …, yt−1). |
| Action a | A decision the agent makes. For a language model, each action is choosing the next token from the vocabulary: at = yt. A full response is a sequence of actions. |
| Policy πθ | The agent's strategy — a function that maps states to a probability distribution over actions. For an LLM, this is the model itself: πθ(at | st) is the probability of generating token at given everything so far. The subscript θ are the model's learnable parameters. |
| Reward r | A scalar signal that tells the agent how good an action (or a completed sequence) was. In RLHF, this comes from a reward model trained on human preferences: "humans preferred output A over output B." Higher reward = more aligned with human intent. |
| Discount γ | A factor in [0, 1] that controls how much the agent cares about future vs. immediate rewards. At γ = 0 it is fully myopic (only the next reward matters); at γ = 1 it weighs all future rewards equally. The discounted return from time t is Gt = rt + γ rt+1 + γ² rt+2 + … In LLM alignment γ is typically close to 1, since we care about the quality of the full response. |
| Value Function V(s) | The expected total (discounted) reward the agent will collect starting from state s and following policy π thereafter: Vπ(s) = 𝔼π[Gt | st = s]. It answers: "how good is it to be in this state?" A separate neural network (the critic) is often trained to approximate this. |
| Advantage A(s, a) | How much better a specific action a is compared to the average action in state s: A(s, a) = Q(s, a) − V(s), where Q(s, a) is the expected return after taking action a in state s. Positive advantage → the action was better than average; negative → worse. This is the core training signal in policy gradient methods. |
| KL Divergence DKL | A measure of how different two probability distributions are. In alignment, DKL[πθ ‖ πref] measures how far the fine-tuned model has drifted from the original pretrained model. It acts as a leash: the policy can improve, but not wander so far that it "forgets" general language ability or exploits reward model quirks (reward hacking). |
| On-policy vs. Off-policy | On-policy methods (PPO, GRPO) generate fresh data from the current policy at each training step — what the model produces right now is what it learns from. Off-policy or offline methods (DPO) learn from a fixed dataset collected earlier, possibly by a different model. On-policy is more expensive (you must run inference every step) but avoids distribution mismatch between training data and the current model. |
With these building blocks in mind, we can now state the problem all three methods are trying to solve.
The Shared Setup
All three methods start from the same RLHF goal: we have a pretrained language model (the reference policy πref) and we want to fine-tune it into an aligned policy πθ that produces outputs humans prefer, while not straying too far from the original model. The core optimization objective (to be maximized) is:
In words: maximize the expected reward of outputs from the current policy, penalized by a KL divergence term that prevents the policy from drifting too far from the reference. The three methods differ in how they solve this problem.
For PPO and GRPO, the loss ℒ is the negative of the clipped surrogate objective (we negate it to turn a maximization into a minimization for the optimizer). For DPO, the loss is already written as a negative log-likelihood to minimize. In all cases: the optimizer calls loss.backward() and steps downhill.
Proximal Policy Optimization
Schulman et al., 2017 · Applied to RLHF by Ouyang et al. (InstructGPT), 2022
What it is
PPO is a general-purpose reinforcement learning algorithm. In the RLHF pipeline, it is used as the optimizer after training a separate reward model rϕ(x, y) on human preference data. The reward model scores outputs, and PPO updates the policy to get higher scores.
The RLHF-PPO Objective — maximize (negate to get loss: ℒ = −J)
where the probability ratio is:
and the advantage is estimated via GAE (Generalized Advantage Estimation) using the reward model and a learned value function Vψ(s):
Term-by-Term Intuition
| ρt(θ) | How much more (or less) likely the new policy is to take action at compared to the old policy that generated the data. If ρ = 1.5, the new policy is 50% more likely to pick this action. |
| clip(ρ, 1−ε, 1+ε) | The trust region mechanism. Caps the ratio so the policy cannot change too aggressively in a single update. Typically ε = 0.2. This is the "proximal" part — stay close to the old policy. |
| At | The advantage: "how much better was this action than the average?" Positive → reinforce the action; negative → suppress it. |
| min(·, ·) | Takes the pessimistic (lower) bound. If the advantage is positive but ρ is too large, clipping kicks in and caps the gradient. If the advantage is negative and ρ is too small, clipping again caps the gradient. This prevents catastrophically large updates. |
| rt | The per-token (or per-step) reward from the reward model, often with a KL penalty baked in: rt = rϕ(x, y) − β·KLt. |
| Vψ(s) | A learned value function (critic) that predicts the expected future return from state s. It serves as a baseline to reduce variance of the advantage estimate. This is a separate neural network that must be trained alongside the policy. |
Direct Preference Optimization
Rafailov et al., 2023
What it is
DPO's key insight is that you can bypass the reward model entirely. The authors show that the optimal solution to the KL-constrained reward maximization problem has a closed-form relationship between the reward and the policy. By substituting this relationship into the Bradley-Terry preference model, you get a loss function that directly trains the policy on preference pairs (yw ≻ yl) — no RL loop, no reward model, no value function.
The Key Reparameterization
The closed-form optimal policy satisfies:
The partition function Z(x) cancels when plugged into the Bradley-Terry model p(yw ≻ yl) = σ(r(x,yw) − r(x,yl)), giving:
The DPO Loss — minimize directly
Term-by-Term Intuition
| σ(·) | The sigmoid function. Maps the reward difference to a probability in [0, 1]. The loss is minimized when this probability is high — i.e., when the model assigns higher implicit reward to the preferred output. |
| log πθ / πref | The implicit reward. How much more the current policy likes output y relative to the reference. This log-ratio acts as a reward signal without needing a separate reward model. It naturally embeds a KL constraint: the reward is measured relative to the reference. |
| β | Controls the strength of the KL constraint. Larger β → more conservative updates (stay closer to πref). Smaller β → more aggressively chase preferences. |
| yw , yl | The winner (preferred) and loser (dispreferred) completions from the preference dataset. DPO needs paired data. |
| −log σ(·) | A cross-entropy–style loss. Gradient pushes up the implicit reward gap: increase the log-ratio for yw, decrease it for yl. |
Group Relative Policy Optimization
Shao et al. (DeepSeek), 2024
What it is
GRPO modifies PPO to eliminate the value function (critic). Instead of training a separate network to estimate advantages, GRPO samples a group of G outputs for each prompt, scores them all with the reward model, and computes advantages by comparing each output's reward to the group mean. This saves the memory and compute of maintaining a critic network the same size as the policy.
The GRPO Objective — maximize (negate to get loss: ℒ = −J)
The group-relative advantage is computed as:
where {y1, …, yG} are G sampled outputs from πθold(·|x) and ri = rϕ(x, yi).
Term-by-Term Intuition
| ri | The reward model's score for the i-th sampled output. Unlike PPO's per-token rewards, GRPO typically uses an outcome-level reward (one score per complete output). |
| mean({r}) | The group mean reward — this is the baseline. It replaces the learned value function Vψ. Conceptually, it answers: "how good is the average output for this prompt?" |
| std({r}) | Normalizes the advantage so it has unit variance. This stabilizes training across prompts that have very different reward scales. |
| Âi | The normalized advantage. Positive for outputs that scored above average in the group; negative for below average. This is a z-score: "how many standard deviations better (or worse) is this output?" |
| clip, ρ, ε | Same PPO-style clipping mechanism for trust region stability. Prevents the policy from changing too fast. |
| β · DKL | An explicit KL penalty against the reference policy. In the DeepSeek formulation, this is computed per-token as an approximation rather than a full KL integral. |
| G | Group size — the number of outputs sampled per prompt. Larger G gives a more stable baseline estimate but costs more compute at generation time. Typical values: G = 8–64. |
Side-by-Side Comparison
| PPO | DPO | GRPO | |
|---|---|---|---|
| Paradigm | RL (actor-critic) | Supervised (contrastive) | RL (actor only) |
| Reward model? | Yes — trained separately | No — implicit in the policy log-ratio | Yes — or rule-based reward |
| Value function? | Yes — same size as policy | No | No — replaced by group statistics |
| Data | Prompts (online generation) | Offline preference pairs (yw, yl) | Prompts (online generation) |
| Advantage | GAE with learned Vψ | N/A (no advantage) | Group-normalized reward z-score |
| KL control | Clipping + optional KL penalty | Implicit via β and log-ratio structure | Clipping + explicit KL penalty |
| Memory cost | 4 models (policy, ref, reward, critic) | 2 models (policy, ref) | 3 models (policy, ref, reward) |
| Key strength | Flexible; proven at scale (InstructGPT, ChatGPT) | Simple; stable; no RL instability | RL flexibility without critic overhead |
| Key weakness | Complex; unstable; expensive (4 models in memory) | Needs pre-collected preference pairs; can underperform on-policy methods | High generation cost (G samples per prompt) |
| Used by | InstructGPT, early ChatGPT, Llama 2 | Zephyr, Intel Neural Chat, many open models | DeepSeek-R1, DeepSeek-Math |
The Big Picture
All three methods solve the same alignment problem. The difference is the path they take:
PPO takes the full RL route: train a reward model, train a critic, then optimize the policy with clipped surrogate gradients. It is maximally flexible (any reward signal works) but heavyweight and tricky to stabilize.
DPO eliminates the RL loop entirely by exploiting the closed-form solution to the KL-constrained optimization. The result is a simple supervised loss on preference pairs. It trades RL's flexibility for stability and simplicity.
GRPO sits between the two: it keeps PPO's on-policy RL framework and reward model but removes the expensive critic by using within-group reward normalization as a baseline. It retains the ability to use any reward signal (including verifiable/rule-based rewards) while being substantially cheaper than full PPO.
When to Use What
If you have clean preference pair data and want simplicity → DPO. If you have a reward signal (learned or rule-based, as in math/code verification) and want on-policy learning → GRPO. If you need maximum flexibility and have the engineering budget → PPO.
References
[1] Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. arxiv.org/abs/1707.06347
[2] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022. arXiv:2203.02155. arxiv.org/abs/2203.02155
[3] Schulman, J., Moritz, P., Levine, S., Jordan, M., & Abbeel, P. (2016). High-Dimensional Continuous Control Using Generalized Advantage Estimation. ICLR 2016. arXiv:1506.02438. arxiv.org/abs/1506.02438
[4] Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. arXiv:2305.18290. arxiv.org/abs/2305.18290
[5] Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., & Guo, D. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300. arxiv.org/abs/2402.03300
[6] DeepSeek-AI, Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., & Zhang, X. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948. arxiv.org/abs/2501.12948
Related & Further Reading
[7] Christiano, P., Leike, J., Brown, T., Marber, M., Legg, S., & Amodei, D. (2017). Deep Reinforcement Learning from Human Preferences. NeurIPS 2017. arXiv:1706.03741. arxiv.org/abs/1706.03741 — The foundational RLHF paper.
[8] Ziegler, D., Stiennon, N., Wu, J., Brown, T., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019). Fine-Tuning Language Models from Human Preferences. arXiv:1909.08593. arxiv.org/abs/1909.08593 — Early application of RLHF to language models.
[9] Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020). Learning to Summarize with Human Feedback. NeurIPS 2020. arXiv:2009.01325. arxiv.org/abs/2009.01325 — PPO-based RLHF for summarization.
[10] Bradley, R. A. & Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4), 324–345. — The preference model underlying DPO's derivation.
[11] Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., & Munos, R. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences. arXiv:2310.12036. arxiv.org/abs/2310.12036 — IPO: a theoretical generalization of DPO.
[12] Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., & Hooker, S. (2024). Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs. ACL 2024. arXiv:2402.14740. arxiv.org/abs/2402.14740 — Compares REINFORCE, PPO, and DPO-family methods.