Tag: ppo

Blog Post·2024-06-19·7 min read

GRPO vs PPO: Why Removing the Value Head Changes Everything

GRPO achieves competitive alignment results without a value function. Here's exactly what changes in the math and implementation, and why that matters for training efficiency and stability.

rl grpo ppo llm-training alignment

Blog Post·2024-06-19·7 min read

The Full PPO Training Step: Value Loss, Entropy, and GAE

ppo-loss-per-token covered the clipped surrogate objective. This post covers what surrounds it: how the value function is trained, where the advantage estimates come from, and why the entropy bonus exists.

rl ppo llm-training rlhf