RLHF (Reinforcement Learning from Human Feedback)

RLHF (Reinforcement Learning from Human Feedback) is the alignment technique that turned raw language models into helpful assistants. Introduced at scale by OpenAI's InstructGPT work in 2022, it fine-tunes a pretrained model so its outputs match what humans actually prefer, rather than merely predicting the statistically next-most-plausible token. A base model completes text; an RLHF-tuned model answers questions, follows instructions, and declines requests — the difference between a library and a librarian.

The three-stage pipeline

Classic RLHF runs in three stages on top of model pretraining. First, supervised fine-tuning (SFT) trains the base model on human-written instruction-and-response examples, teaching the assistant format. Second, human labelers rank several model outputs for the same prompt, and those rankings train a separate reward model that learns to score any response with a scalar — a learned stand-in for human judgment that can grade millions of outputs no labeling team could review. Third, the language model is optimized against that reward model with a reinforcement-learning algorithm, classically PPO, which constrains each update so training stays stable. A KL-divergence penalty against the SFT model keeps the tuned policy from drifting into gibberish that happens to please the reward model.

What can go wrong

The reward model is a proxy, and proxies get gamed. Over-optimized models learn to exploit its blind spots — a failure called reward hacking — producing confident, padded, agreeable text that scores well without being better. Related is the tendency toward sycophancy: telling users what they want to hear, because agreement rated well in training. These are not incidental bugs but the central difficulty of the method, and they explain the KL leash, early stopping, and the constant search for better reward signals.

Cost, and the successors it motivated

RLHF is computationally heavy because stage three juggles several large models at once — policy, reference, reward model, and PPO's value network. That burden motivated simpler successors: DPO optimizes on preference pairs directly with no reinforcement-learning loop or reward model, critic-free methods like GRPO cut the value network, and AI-feedback variants such as Constitutional AI replace much of the human labeling with model-generated critiques guided by written principles. Most modern open-weight assistants are aligned with some blend of these cheaper descendants rather than textbook RLHF.

Why it matters for sovereignty

RLHF is why assistant models behave consistently — and also how they inherit the preferences, policies, and refusal boundaries of whoever ran the training. When you download an aligned open-weight model, you are downloading someone's judgment calls baked into the weights, invisible in any config file. Understanding the pipeline clarifies what is and is not fixed: further fine-tuning can shift an aligned model's behavior, which is exactly what practitioners with their own hardware and their own values do. RLHF sits within the broader project of AI alignment; for the self-hoster, it is also a reminder that "neutral" models do not exist — only models whose training choices you have or have not examined.

A useful habit when evaluating any downloaded model is to read its model card for the alignment recipe: whether it saw RLHF, DPO, or a blend, and whose preference data drove it. Two models with identical architectures and near-identical benchmark scores can behave very differently under pressure — one hedging and refusing where the other answers plainly — and the divergence traces back to this stage of training, not to the weights' size or the context window. Knowing the recipe will not tell you everything, but it tells you what questions to test before you trust a model with real work. The recipe is as much a part of the model as the parameter count, and treating it that way is basic due diligence for anyone running models on their own hardware.