RLHF

A base language model can produce fluent text, but fluency alone does not tell it which answers people actually find helpful, honest, or safe. It imitates patterns from its training data, yet it has no direct signal for human preferences. To align its behavior with what people want, you need feedback that reflects human taste.

RLHF, reinforcement learning from human feedback, adds that signal through a simple loop. Given a prompt, the model generates several candidate responses. Humans compare them and choose which is better. Those comparisons become training data that pushes the model toward responses people prefer.

The preference has to become something a model can optimize. First, the human comparisons train a separate reward model: it takes a prompt and a response and outputs a score predicting how a human would rank it. Then the language model is fine-tuned to produce responses that score well under that reward model. The reward model stands in for the human during training, so a person does not have to judge every new output.

Over many rounds, behavior shifts. The model produces more of the traits that win comparisons, like being clearer, more on topic, and more careful, and less of the traits that lose, like being evasive, unsafe, or misleading.

The figure below is a simplified stand-in. It uses a few hidden trait scores (helpful, honest, concise, and enthusiastic) and a small preference rule, not a real language model, but the loop is the real idea: generate, compare, learn the preference, shift behavior. Pick the better response each round and watch the model's preference move.

Picking the better response

Fine-tuning, more generally

RLHF is one example of a broader technique called fine-tuning. Fine-tuning takes a model that has already been trained on a huge, general corpus and trains it a little further on a smaller, specialized dataset. The goal is to shift the model's behavior toward a particular style, domain, or set of preferences, without starting over from scratch.

The starting point is what makes this work. The base model already knows grammar and facts from its original training. Following instructions is not part of that: it is taught afterwards, first by supervised fine-tuning (SFT) on example instructions with good answers. The full pipeline for an assistant is SFT, then a reward model trained on human comparisons, then RLHF, which starts from the SFT model, not the raw base model. Fine-tuning does not rebuild any of that; it nudges the model on top of what it already knows, which is why it needs far less data and compute than training a model from the beginning. RLHF fits this pattern exactly: the specialized dataset is human preference comparisons, and the further training pushes the model toward responses people prefer.

A common use of fine-tuning is teaching a general assistant a specific domain, like a company's support style. You collect a small set of example questions and good answers, then train on them. Afterward the model answers domain questions in the specialized way, while keeping its general knowledge intact. Done carefully, specializing behavior does not erase what the base model already knew.

The figure below shows this general idea. A base model answers some prompts generically. Run fine-tuning on a small labeled domain dataset and watch the domain answers specialize, while a couple of general-knowledge answers stay the same. As before, the responses are hand-authored to illustrate the behavior, not produced by real training.