Researchers have identified two measurable properties of preference training data that reliably predict whether a language model will improve its reasoning abilities — a finding that may reshape how AI developers construct alignment datasets.

Preference optimization — the process of teaching a language model to favour certain outputs over others — has become a cornerstone of modern AI alignment. Techniques such as DPO (Direct Preference Optimization) and KTO are now standard tools for steering model behaviour after initial training. Yet until now, practitioners have had limited scientific guidance on what actually makes one preference dataset better than another. The new paper, posted to ArXiv CS.CL under the title Decomposing the Delta, sets out to address that gap.

Two Types of Quality Gap — and Why They're Different

The research centres on what the authors call the "quality delta" in preference data — the difference in quality between the "chosen" (preferred) and "rejected" (dispreferred) examples that form each training pair. Critically, the study separates this into two distinct concepts that previous work had conflated.

The first is generator-level delta: the capability gap between the models used to produce the chosen and rejected reasoning traces. A high generator-level delta means the good examples come from a substantially more capable model than the bad ones. The second is sample-level delta: within a single training pair, how different in quality are the chosen and rejected outputs, as judged by an independent evaluator?

Increasing generator-level delta steadily improves performance on out-of-domain reasoning tasks — a finding that holds across model families and scales.

To measure sample-level delta, the researchers used an LLM-as-a-judge approach, rating generated reasoning traces across multiple quality dimensions. This is a self-reported benchmark methodology — the quality scores depend on the judging model's own assessments, which introduces a layer of subjectivity worth noting.

What the Experiments Found

The team systematically varied both types of delta across experiments. For generator-level delta, they tested models of different scales and from different model families to produce the chosen examples, while keeping the rejected examples consistent. The results were consistent: a larger capability gap between generators produced steadily better reasoning performance on tasks outside the training distribution — a strong signal of genuine generalisation rather than overfitting.

For sample-level delta, the findings were more nuanced but practically useful. Filtering training data to include only pairs with a high within-pair quality difference did not necessarily improve peak performance, but it did enable more efficient training — meaning models reached comparable performance with less data. For teams working under compute or data-budget constraints, this is a directly actionable result.

The study's out-of-domain evaluation is particularly significant. Many alignment papers test models on tasks similar to their training data, which can flatter results. Testing on genuinely different reasoning tasks provides stronger evidence that the improvements reflect a real capability gain.

A Practical Recipe for Dataset Construction

The paper distills its findings into a two-step recommendation for practitioners building preference datasets for reasoning tasks. First, maximise generator-level delta by using the most capable available model to generate chosen examples, while using a substantially weaker model for rejected examples. The greater the capability gap, the more signal each training pair carries.

Second, exploit sample-level delta during data selection. Once a dataset is generated, filtering to retain only pairs where the quality difference is large — as assessed by an LLM judge — allows developers to train on a smaller, more informative subset without sacrificing performance. This two-stage approach could reduce the cost of preference data curation significantly.

The research does not specify which model families or scales were used in every configuration, and the reliance on LLM-as-a-judge scoring means the quality ratings carry the biases and limitations of whichever model performs the evaluation. Independent replication across different judge models would strengthen the findings.

Implications for the Alignment Pipeline

The paper arrives at a moment when the AI industry is investing heavily in post-training alignment, with preference optimization sitting at the centre of that effort. Companies and research labs generating synthetic preference data — an increasingly common approach as human annotation costs rise — now have empirical guidance suggesting that the source model's capability matters as much as the volume of data produced.

This challenges a common assumption in dataset scaling: that more preference pairs are always better. The study suggests that fewer, higher-delta pairs outperform larger datasets of low-delta pairs in terms of training efficiency. That reframing has direct cost implications for anyone running large-scale alignment pipelines.

The work also adds scientific weight to the observation that preference data quality is not a single dimension. Generator-level and sample-level delta behave differently and should be managed separately — a distinction that existing data curation tools and pipelines do not always make explicit.

What This Means

For AI developers building or refining alignment datasets, this research provides an empirical framework for what makes preference data effective: pair a strong model against a weak one, then filter by within-pair quality difference to achieve more reasoning improvement per training example.