Iterative DPO

Iterative DPO is the practice of running Direct Preference Optimization not once over a fixed dataset, but in repeated rounds, where fresh preference pairs are sampled from the most recent version of the model before each round. Train, regenerate data from the improved model, relabel, train again. The loop keeps the training data close to what the model currently produces, and that single change addresses the most consequential weakness of one-shot preference tuning.

The distribution-gap problem

Plain offline DPO learns from a static preference dataset whose chosen and rejected responses were typically written by some other model — an earlier checkpoint, a different model family, or a mix scraped from public sources. As training proceeds and the policy improves, its own outputs drift steadily away from that frozen snapshot. The model is then being graded on essays it would never write: the preference signal describes a distribution it has left behind. This mismatch — off-policy data, in reinforcement-learning terms — can flatten out gains and, in the worse case, push probability mass toward out-of-distribution responses the static data never covered. The DPO loss also anchors to a fixed reference model, and as the policy travels far from that reference, the anchor's regularization becomes less meaningful.

The iterative loop

Iterative DPO fixes both issues mechanically. Each round proceeds in four steps: sample multiple candidate responses per prompt from the current policy; label pairs as chosen versus rejected using a judge — a trained reward model, a stronger LLM prompted as evaluator, or human annotators; run a round of DPO on the fresh pairs; then promote the result to become the generator (and typically the new reference model) for the next round. Because every round's data is drawn from the model's own current behavior, the preference signal always lands on-policy — the model learns from feedback on text it would actually generate. In practice a few rounds capture most of the benefit, with returns diminishing as the judge's ability to distinguish ever-better responses becomes the bottleneck. The round structure also gives you natural checkpoints: every round ends with a complete model and a complete dataset, so a bad round can be diagnosed, discarded, and rerun without unwinding a long training run.

Self-rewarding loops

The most striking application is the Self-Rewarding Language Models framework, where a single model plays both roles: it generates candidate responses and judges them via LLM-as-a-judge prompting, then trains on its own self-labeled pairs through iterative DPO. Each round improves not only the model's answers but — because judging is itself a language task — its judging, letting alignment bootstrap without an external annotation vendor in the loop. The honest caveat: a self-judging loop can amplify the judge's biases along with its skills, so periodic external evaluation remains the guard rail against a model that grades its own homework ever more generously.

Why it fits sovereign AI

For a self-hoster aligning models on owned hardware, iterative DPO is one of the most practical paths to continual improvement. It needs no reinforcement-learning machinery — each round is ordinary DPO training — and generation between rounds is embarrassingly parallel inference you can schedule on idle GPU hours. The annotation burden that makes preference tuning expensive is replaced by a judge you run locally, keeping both the data and the values it encodes under your control: nothing about your prompts, your domain, or your preferences leaves the building. Where fully continuous Online DPO folds generation, labeling, and updating into one streaming loop, iterative DPO delivers most of the same on-policy benefit in discrete, restartable, debuggable rounds — a better operational fit for a small rig than a heavyweight training pipeline. The pattern generalizes: sample, judge, train, repeat — compounding capability on hardware you own.