# Preference tuning Teach a model which of two answers to prefer, with DPO or KTO, from a file of preference pairs. Preference tuning trains a model on pairs of answers to the same prompt: one preferred, one not. Use it when judging which of two answers is better is easier than writing the ideal one, for example to fix tone, refusals or a house style. You run it with the same [`pm.finetune`](https://docs.privatemind.com/sdk-finetune.html) call as supervised fine-tuning, with `task="dpo"` or `task="kto"`. Methods, checkpoints, resources, registry weights and tracking work as the [Fine-tuning](https://docs.privatemind.com/sdk-finetune.html) page describes. This page covers what is different. It needs SDK 0.11.0 or later. ## Run it ```python import privatemind as pm run = pm.finetune( model="Qwen/Qwen3-8B", dataset="/workspace/data/prefs.jsonl", output_path="/workspace/adapters/support-dpo", task="dpo", method="lora", gpus_per_worker=8, ) run.wait() print(run.phase, run.failure_reason) ``` Preference tuning works best on a model that already answers in the format you want, such as an instruction-tuned model or the result of an SFT run. To start from an SFT result, [publish it](https://docs.privatemind.com/sdk-evaluate.html#publish-a-fine-tune) and pass `weights=":"` in place of `model=`. ## DPO or KTO Both take the same files. - **DPO** (Direct Preference Optimization) learns from each pair as a pair. It raises how likely the preferred answer is compared with the non-preferred one, measured against the model it started from. - **KTO** (Kahneman-Tversky Optimization) splits each pair and learns from every answer on its own, as desirable or undesirable. ## Data format One pair per line, in either of two shapes. The SDK detects the shape from the first row and checks the first 50 rows before it submits, as for SFT. In the job, both shapes become the form the trainer uses. The `preference` format is the shape OpenAI and Together use for preference fine-tuning: ```json {"input": {"messages": [{"role": "user", "content": "Summarise our refund policy."}]}, "preferred_output": [{"role": "assistant", "content": "Refunds are open for 30 days from delivery, with proof of purchase."}], "non_preferred_output": [{"role": "assistant", "content": "We have a refund policy."}]} ``` - `input.messages`: the conversation so far, ending with the turn the model answers. It may start with a system message. - `preferred_output` and `non_preferred_output`: each a list holding one assistant message. The `prompt_chosen_rejected` format is the shape TRL, the training library, uses: ```json {"prompt": [{"role": "user", "content": "Summarise our refund policy."}], "chosen": [{"role": "assistant", "content": "Refunds are open for 30 days from delivery, with proof of purchase."}], "rejected": [{"role": "assistant", "content": "We have a refund policy."}]} ``` The three fields are either all lists of messages, as here, or all plain strings. `prompt` may be left out: the trainer then takes the part `chosen` and `rejected` start with in common as the prompt. Keep both answers in a pair to the same prompt and about the same length where you can. A model trained on pairs where the preferred answer is always longer learns to write longer answers. ## Settings The fields in [Method and hyperparameters](https://docs.privatemind.com/sdk-finetune.html#method-and-hyperparameters) apply, with two differences: `lr` defaults to `5e-6`, or `1e-6` for `method="full"`, and `packing` has no effect. Two more fields go to the trainer: | Field | DPO | KTO | |---|---|---| | `loss_type` | `sigmoid` by default; also `hinge`, `ipo`, `robust`, `apo_zero` and the other DPO losses the SDK names when it refuses one | `kto` by default, or `apo_zero_unpaired` | | `beta` | how strongly the model is held to the base model; the trainer's default unless you set it | the same | The SDK refuses a `loss_type` the task doesn't take, and names the ones it does. ## What to watch The trainer logs reward figures to MLflow at every logging step. For DPO, `rewards/margins` is how far the preferred answers score above the non-preferred ones, and `rewards/accuracies` the share of pairs where the preferred answer scores higher. Both should rise as training goes on. Pass `eval_dataset=` with held-out pairs in the same shape to get the same figures on data the model hasn't trained on. ## Where next - [Evaluate and publish](https://docs.privatemind.com/sdk-evaluate.html): score the tuned model on your own eval set, push it to your registry and serve it. - [Reinforcement learning](https://docs.privatemind.com/sdk-grpo.html): train against a reward function in place of fixed pairs. - [Fine-tuning](https://docs.privatemind.com/sdk-finetune.html): methods, hyperparameters, checkpoints and resources.