GRPO (Group Relative Policy Optimization) is reinforcement learning for language models. For each prompt, the model writes several answers. A reward scores each one, and training moves the model towards the answers that scored above the others in their group. You need prompts and a way to score an answer, and no example answers. It suits tasks with a result you can check in code: a number, a label, valid JSON, a fixed format.
You run it with pm.finetune and task="grpo". Methods, checkpoints, registry weights and tracking work as the Fine-tuning page describes. It needs SDK 0.11.0 or later.
Run it
import privatemind as pm
run = pm.finetune(
model="Qwen/Qwen3-8B",
dataset="data/arithmetic.jsonl",
output_path="/workspace/adapters/arithmetic-grpo",
task="grpo",
reward="string_check",
gpus_per_worker=8,
)
run.wait()reward="string_check" scores an answer 1.0 when it matches the row's answer field and 0.0 otherwise. For anything else, write a reward function.
Data format
One prompt per line, as messages, plus any fields your reward needs:
{"messages": [{"role": "user", "content": "What is 17 * 23? Reply with the number only."}], "answer": "391"}messages: the conversation the model answers. It may start with a system message, and it can't end with an assistant turn.- Any other field: passed to the reward, unchanged, as part of
item.
Before it submits, the SDK checks the rows it samples against these rules, and with string_check that each has the reference field.
Reward functions
reward takes a Python function with the signature OpenAI uses for its graders:
def grade(sample, item) -> float:
return 1.0 if sample["output_text"].strip() == item["answer"] else 0.0sampleis one answer the model wrote.sample["output_text"]is its text, andsample["output_json"]the text parsed as JSON, orNonewhen it isn't JSON.itemis the dataset row the prompt came from, with all its fields.- The function returns a float: higher means a better answer. GRPO compares answers to the same prompt with each other, so use one scale for every prompt, such as 0 to 1.
It is a plain function, so try it in the notebook on a row before you submit:
row = {"messages": [...], "answer": "391"}
grade({"output_text": "391"}, row) # 1.0
grade({"output_text": "390"}, row) # 0.0The SDK ships your function to the job the way @pm.train ships one: it serialises it with cloudpickle, copies it to your home Volume and loads it in the job. It runs inside your own job, on your own allocation, with the job's access. Outside a workspace, install the extra that does the serialising: pip install 'privatemind[promote]'. The same rules apply:
- Keep it self-contained. Read everything from
sampleanditem. A notebook global the function uses either fails the submit or travels with it. - Import inside the function, and only what the training image provides.
- Keep it fast. It runs once for every answer the model writes, and training waits for the scores.
A function only loads on the Python and cloudpickle versions it was serialised with. The job checks both before it starts and stops with a message naming them if they differ. Submit from a workspace whose versions match, or use string_check, which needs no shipping.
Exact match: string_check
reward="string_check" scores 1.0 when the answer, with whitespace removed from both ends, equals the row's answer field, trimmed the same way. Anything else, and a row without the field, scores 0.0. To compare against another field, pass reference_field=:
run = pm.finetune(
model="Qwen/Qwen3-8B",
dataset="data/labels.jsonl",
output_path="/workspace/adapters/labels-grpo",
task="grpo",
reward="string_check",
reference_field="label",
gpus_per_worker=8,
)Thinking models
The reward scores only the final answer. When an answer contains a <think>...</think> block, as Qwen3's do, everything up to the end of the block is cut before the answer is scored. To have the model answer without thinking, pass chat_template_kwargs={"enable_thinking": False}.
Settings
| Field | Default | What it does |
|---|---|---|
num_generations |
8 |
answers the model writes per prompt, 2 to 256 |
max_completion_length |
1024 |
token cap on each answer |
loss_type |
dapo |
the trainer's loss: dapo, grpo, dr_grpo, bnpo, cispo, sapo or vespo |
beta |
the trainer's default | how strongly the model is held to the base model |
lr |
1e-5, or 1e-6 for method="full" |
learning rate |
per_device_batch_size × gradient_accumulation_steps × GPUs must be a multiple of num_generations. The SDK checks this before it submits. The defaults, 4 × 4 on 8 GPUs with 8 answers per prompt, fit.
How it runs
GRPO runs on one node, on the training-rl training image, which carries vLLM. The SDK picks that image when you leave image unset. vLLM runs inside the training processes on the same GPUs and keeps part of each GPU's memory. It generates the answers for a step and takes the new weights straight from the trainer.
What to watch
The trainer logs the mean reward of each step to MLflow. It should rise as the model learns. If it stays flat at 0, the model rarely gets an answer right: check the reward on a few answers from the base model, or start from an easier set of prompts.
Where next
- Evaluate and publish: score the tuned model with the same kind of function, push it to your registry and serve it.
- Preference tuning: DPO and KTO, when you have pairs of answers in place of a reward.
- Fine-tuning: methods, hyperparameters, checkpoints and resources.