SimPO (Simple Preference Optimization)

SimPO (Simple Preference Optimization) is a reference-free method for aligning language models to human preferences. Introduced by Meng, Xia, and Chen in 2024, it defines the implicit reward as the average log-probability of a response — that is, length-normalized — and adds a target reward margin to push winning and losing responses further apart.

Length normalization and the margin

Earlier direct methods reward the total log-probability of a sequence, which can quietly favor longer or shorter outputs and break the link between training reward and what the model actually generates. By dividing by length, SimPO keeps its reward ranking consistent with generation-time likelihood. The added margin term, borrowed from the Bradley-Terry preference model, encourages a clear gap between the chosen and rejected response rather than a razor-thin one.

Efficiency without a reference model

Like ORPO, SimPO drops the frozen reference model that direct preference optimization relies on, cutting compute and memory. The authors reported that SimPO outperformed direct preference optimization on benchmarks such as AlpacaEval 2 and Arena-Hard without inflating response length — a common side effect of reward optimization known as length bias.

For a self-hosted alignment workflow, SimPO's appeal is doing more with a single model in memory and a reward that does not drift toward verbosity. See the related reference-free approach ORPO (Odds Ratio Preference Optimization) and the binary-feedback method KTO (Kahneman-Tversky Optimization).