# Reinforcement learning Train a model with GRPO: it writes answers to your prompts, a reward scores them, and the model learns from the scores. GRPO (Group Relative Policy Optimization) is reinforcement learning for language models. For each prompt, the model writes several answers. A reward scores each one, and training moves the model towards the answers that scored above the others in their group. You need prompts and a way to score an answer, and no example answers. It suits tasks with a result you can check in code: a number, a label, valid JSON, a fixed format. You run it with [`pm.finetune`](https://docs.privatemind.com/sdk-finetune.html) and `task="grpo"`. Methods, checkpoints, registry weights and tracking work as the Fine-tuning page describes. It needs SDK 0.11.0 or later. ## Run it ```python import privatemind as pm run = pm.finetune( model="Qwen/Qwen3-8B", dataset="data/arithmetic.jsonl", output_path="/workspace/adapters/arithmetic-grpo", task="grpo", reward="string_check", gpus_per_worker=8, ) run.wait() ``` `reward="string_check"` scores an answer 1.0 when it matches the row's `answer` field and 0.0 otherwise. For anything else, write a [reward function](https://docs.privatemind.com/sdk-grpo.html#reward-functions). ## Data format One prompt per line, as `messages`, plus any fields your reward needs: ```json {"messages": [{"role": "user", "content": "What is 17 * 23? Reply with the number only."}], "answer": "391"} ``` - `messages`: the conversation the model answers. It may start with a system message, and it can't end with an assistant turn. - Any other field: passed to the reward, unchanged, as part of `item`. Before it submits, the SDK checks the rows it samples against these rules, and with `string_check` that each has the reference field. ## Reward functions `reward` takes a Python function with the signature OpenAI uses for its graders: ```python def grade(sample, item) -> float: return 1.0 if sample["output_text"].strip() == item["answer"] else 0.0 ``` - `sample` is one answer the model wrote. `sample["output_text"]` is its text, and `sample["output_json"]` the text parsed as JSON, or `None` when it isn't JSON. - `item` is the dataset row the prompt came from, with all its fields. - The function returns a float: higher means a better answer. GRPO compares answers to the same prompt with each other, so use one scale for every prompt, such as 0 to 1. It is a plain function, so try it in the notebook on a row before you submit: ```python row = {"messages": [...], "answer": "391"} grade({"output_text": "391"}, row) # 1.0 grade({"output_text": "390"}, row) # 0.0 ``` The SDK ships your function to the job the way [`@pm.train`](https://docs.privatemind.com/training-sdk.html#promote-a-notebook-function) ships one: it serialises it with cloudpickle, copies it to your home Volume and loads it in the job. It runs inside your own job, on your own allocation, with the job's access. Outside a workspace, install the extra that does the serialising: `pip install 'privatemind[promote]'`. The same rules apply: 1. **Keep it self-contained.** Read everything from `sample` and `item`. A notebook global the function uses either fails the submit or travels with it. 2. **Import inside the function**, and only what the training image provides. 3. **Keep it fast.** It runs once for every answer the model writes, and training waits for the scores. A function only loads on the Python and cloudpickle versions it was serialised with. The job checks both before it starts and stops with a message naming them if they differ. Submit from a workspace whose versions match, or use `string_check`, which needs no shipping. ### Exact match: `string_check` `reward="string_check"` scores 1.0 when the answer, with whitespace removed from both ends, equals the row's `answer` field, trimmed the same way. Anything else, and a row without the field, scores 0.0. To compare against another field, pass `reference_field=`: ```python run = pm.finetune( model="Qwen/Qwen3-8B", dataset="data/labels.jsonl", output_path="/workspace/adapters/labels-grpo", task="grpo", reward="string_check", reference_field="label", gpus_per_worker=8, ) ``` ### Thinking models The reward scores only the final answer. When an answer contains a `...` block, as Qwen3's do, everything up to the end of the block is cut before the answer is scored. To have the model answer without thinking, pass `chat_template_kwargs={"enable_thinking": False}`. ## Settings | Field | Default | What it does | |---|---|---| | `num_generations` | `8` | answers the model writes per prompt, `2` to `256` | | `max_completion_length` | `1024` | token cap on each answer | | `loss_type` | `dapo` | the trainer's loss: `dapo`, `grpo`, `dr_grpo`, `bnpo`, `cispo`, `sapo` or `vespo` | | `beta` | the trainer's default | how strongly the model is held to the base model | | `lr` | `1e-5`, or `1e-6` for `method="full"` | learning rate | `per_device_batch_size` × `gradient_accumulation_steps` × GPUs must be a multiple of `num_generations`. The SDK checks this before it submits. The defaults, 4 × 4 on 8 GPUs with 8 answers per prompt, fit. ## How it runs GRPO runs on one node, on the `training-rl` training image, which carries vLLM. The SDK picks that image when you leave `image` unset. vLLM runs inside the training processes on the same GPUs and keeps part of each GPU's memory. It generates the answers for a step and takes the new weights straight from the trainer. ## What to watch The trainer logs the mean reward of each step to MLflow. It should rise as the model learns. If it stays flat at 0, the model rarely gets an answer right: check the reward on a few answers from the base model, or start from an easier set of prompts. ## Where next - [Evaluate and publish](https://docs.privatemind.com/sdk-evaluate.html): score the tuned model with the same kind of function, push it to your registry and serve it. - [Preference tuning](https://docs.privatemind.com/sdk-preference-tuning.html): DPO and KTO, when you have pairs of answers in place of a reward. - [Fine-tuning](https://docs.privatemind.com/sdk-finetune.html): methods, hyperparameters, checkpoints and resources.