Preference tuning trains a model on pairs of answers to the same prompt: one preferred, one not. Use it when judging which of two answers is better is easier than writing the ideal one, for example to fix tone, refusals or a house style. You run it with the same pm.finetune call as supervised fine-tuning, with task="dpo" or task="kto".
Methods, checkpoints, resources, registry weights and tracking work as the Fine-tuning page describes. This page covers what is different. It needs SDK 0.11.0 or later.
Run it
import privatemind as pm
run = pm.finetune(
model="Qwen/Qwen3-8B",
dataset="/workspace/data/prefs.jsonl",
output_path="/workspace/adapters/support-dpo",
task="dpo",
method="lora",
gpus_per_worker=8,
)
run.wait()
print(run.phase, run.failure_reason)Preference tuning works best on a model that already answers in the format you want, such as an instruction-tuned model or the result of an SFT run. To start from an SFT result, publish it and pass weights="<name>:<version>" in place of model=.
DPO or KTO
Both take the same files.
- DPO (Direct Preference Optimization) learns from each pair as a pair. It raises how likely the preferred answer is compared with the non-preferred one, measured against the model it started from.
- KTO (Kahneman-Tversky Optimization) splits each pair and learns from every answer on its own, as desirable or undesirable.
Data format
One pair per line, in either of two shapes. The SDK detects the shape from the first row and checks the first 50 rows before it submits, as for SFT. In the job, both shapes become the form the trainer uses.
The preference format is the shape OpenAI and Together use for preference fine-tuning:
{"input": {"messages": [{"role": "user", "content": "Summarise our refund policy."}]}, "preferred_output": [{"role": "assistant", "content": "Refunds are open for 30 days from delivery, with proof of purchase."}], "non_preferred_output": [{"role": "assistant", "content": "We have a refund policy."}]}input.messages: the conversation so far, ending with the turn the model answers. It may start with a system message.preferred_outputandnon_preferred_output: each a list holding one assistant message.
The prompt_chosen_rejected format is the shape TRL, the training library, uses:
{"prompt": [{"role": "user", "content": "Summarise our refund policy."}], "chosen": [{"role": "assistant", "content": "Refunds are open for 30 days from delivery, with proof of purchase."}], "rejected": [{"role": "assistant", "content": "We have a refund policy."}]}The three fields are either all lists of messages, as here, or all plain strings. prompt may be left out: the trainer then takes the part chosen and rejected start with in common as the prompt.
Keep both answers in a pair to the same prompt and about the same length where you can. A model trained on pairs where the preferred answer is always longer learns to write longer answers.
Settings
The fields in Method and hyperparameters apply, with two differences: lr defaults to 5e-6, or 1e-6 for method="full", and packing has no effect. Two more fields go to the trainer:
| Field | DPO | KTO |
|---|---|---|
loss_type |
sigmoid by default; also hinge, ipo, robust, apo_zero and the other DPO losses the SDK names when it refuses one |
kto by default, or apo_zero_unpaired |
beta |
how strongly the model is held to the base model; the trainer's default unless you set it | the same |
The SDK refuses a loss_type the task doesn't take, and names the ones it does.
What to watch
The trainer logs reward figures to MLflow at every logging step. For DPO, rewards/margins is how far the preferred answers score above the non-preferred ones, and rewards/accuracies the share of pairs where the preferred answer scores higher. Both should rise as training goes on. Pass eval_dataset= with held-out pairs in the same shape to get the same figures on data the model hasn't trained on.
Where next
- Evaluate and publish: score the tuned model on your own eval set, push it to your registry and serve it.
- Reinforcement learning: train against a reward function in place of fixed pairs.
- Fine-tuning: methods, hyperparameters, checkpoints and resources.