Get API key

Preference tuning

View as Markdown

Teach a model which of two answers to prefer, with DPO or KTO, from a file of preference pairs.

Preference tuning trains a model on pairs of answers to the same prompt: one preferred, one not. Use it when judging which of two answers is better is easier than writing the ideal one, for example to fix tone, refusals or a house style. You run it with the same pm.finetune call as supervised fine-tuning, with task="dpo" or task="kto".

Methods, checkpoints, resources, registry weights and tracking work as the Fine-tuning page describes. This page covers what is different. It needs SDK 0.11.0 or later.

Run it

Python
import privatemind as pm

run = pm.finetune(
    model="Qwen/Qwen3-8B",
    dataset="/workspace/data/prefs.jsonl",
    output_path="/workspace/adapters/support-dpo",
    task="dpo",
    method="lora",
    gpus_per_worker=8,
)

run.wait()
print(run.phase, run.failure_reason)

Preference tuning works best on a model that already answers in the format you want, such as an instruction-tuned model or the result of an SFT run. To start from an SFT result, publish it and pass weights="<name>:<version>" in place of model=.

DPO or KTO

Both take the same files.

  • DPO (Direct Preference Optimization) learns from each pair as a pair. It raises how likely the preferred answer is compared with the non-preferred one, measured against the model it started from.
  • KTO (Kahneman-Tversky Optimization) splits each pair and learns from every answer on its own, as desirable or undesirable.

Data format

One pair per line, in either of two shapes. The SDK detects the shape from the first row and checks the first 50 rows before it submits, as for SFT. In the job, both shapes become the form the trainer uses.

The preference format is the shape OpenAI and Together use for preference fine-tuning:

JSON
{"input": {"messages": [{"role": "user", "content": "Summarise our refund policy."}]}, "preferred_output": [{"role": "assistant", "content": "Refunds are open for 30 days from delivery, with proof of purchase."}], "non_preferred_output": [{"role": "assistant", "content": "We have a refund policy."}]}
  • input.messages: the conversation so far, ending with the turn the model answers. It may start with a system message.
  • preferred_output and non_preferred_output: each a list holding one assistant message.

The prompt_chosen_rejected format is the shape TRL, the training library, uses:

JSON
{"prompt": [{"role": "user", "content": "Summarise our refund policy."}], "chosen": [{"role": "assistant", "content": "Refunds are open for 30 days from delivery, with proof of purchase."}], "rejected": [{"role": "assistant", "content": "We have a refund policy."}]}

The three fields are either all lists of messages, as here, or all plain strings. prompt may be left out: the trainer then takes the part chosen and rejected start with in common as the prompt.

Keep both answers in a pair to the same prompt and about the same length where you can. A model trained on pairs where the preferred answer is always longer learns to write longer answers.

Settings

The fields in Method and hyperparameters apply, with two differences: lr defaults to 5e-6, or 1e-6 for method="full", and packing has no effect. Two more fields go to the trainer:

Field DPO KTO
loss_type sigmoid by default; also hinge, ipo, robust, apo_zero and the other DPO losses the SDK names when it refuses one kto by default, or apo_zero_unpaired
beta how strongly the model is held to the base model; the trainer's default unless you set it the same

The SDK refuses a loss_type the task doesn't take, and names the ones it does.

What to watch

The trainer logs reward figures to MLflow at every logging step. For DPO, rewards/margins is how far the preferred answers score above the non-preferred ones, and rewards/accuracies the share of pairs where the preferred answer scores higher. Both should rise as training goes on. Pass eval_dataset= with held-out pairs in the same shape to get the same figures on data the model hasn't trained on.

Where next