# Evaluate and publish Push a fine-tune to your org's registry, score it on your own eval set, and serve it through a model server. A fine-tune ends with a result on your Volume: an adapter, or a whole model. Three steps take it from there to something your applications call. `run.publish` pushes it to your org's [registry](https://docs.privatemind.com/registry.html) as a model version. `pm.evaluate` scores a model on an eval set of your own. A [model server](https://docs.privatemind.com/sdk-model-servers.html) then serves the version you choose. All three need SDK 0.11.0 or later, and publishing needs the registry turned on for your org. Run `publish` and `evaluate` in a workspace on the home Volume your fine-tunes use: both stage files there and read results back from it. ## Publish a fine-tune ```python import privatemind as pm run = pm.get_job("ft-support-dpo") # or the Run that pm.finetune returned run.wait() ref = run.publish("support-bot-dpo") print(ref) # support-bot-dpo:1 ``` `publish(name, version=None)` takes a model name with the [registry's naming rules](https://docs.privatemind.com/sdk-weights.html#push) and pushes the result as a version of that model. It returns the reference, `:`. Leave `version` out and it is one more than the model's highest numeric version, so `1` for a new model. For a LoRA or QLoRA run, `publish` first merges the adapter into the base model, in a one-GPU job on the same cluster, and waits for it. The version is then a complete model that needs nothing else to load. A full-parameter run is pushed as trained. `publish` takes a run that succeeded, which `pm.finetune` submitted from this home Volume in the last 14 days, and raises `ValidationError` for any other. The new version is an ordinary registry version: list it, pull it, share it or delete it like any other. ## Evaluate a model `pm.evaluate` answers every prompt in an eval file with the model and scores each answer with a grader: ```python result = pm.evaluate( weights="support-bot-dpo:1", dataset="data/eval.jsonl", grader="string_check", ) print(result.mean, len(result.scores)) ``` It runs as a [training job](https://docs.privatemind.com/training-sdk.html) on your GPU allocation, on the `training-rl` image, and waits for it. vLLM writes one answer to each prompt, and the grader scores each answer. `evaluate` returns an `EvalResult`: `scores` holds one score per row, in the file's order, `mean` their mean, and `run` the job. If the job doesn't succeed, `evaluate` raises `PrivatemindError` with its failure message. The mean and the row count also go to MLflow when your org has it set up. The eval file has the same shape as a [GRPO dataset](https://docs.privatemind.com/sdk-grpo.html#data-format): one prompt per line as `messages`, plus any fields the grader needs. The grader works as a [GRPO reward](https://docs.privatemind.com/sdk-grpo.html#reward-functions) does: `"string_check"` against the row's `answer` field, or a `grade(sample, item) -> float` function of your own. So the same function can train a model with GRPO and then measure it. A `...` block is cut from each answer before it is graded. | Argument | Default | What it does | |---|---|---| | `weights` or `model` | required, one of them | a `:` from your registry, or a Hugging Face id or Volume path | | `dataset` | required | the eval file, under your home Volume | | `grader` | required | `"string_check"`, or a `grade(sample, item)` function | | `reference_field` | `"answer"` | the field `string_check` compares with | | `gpus` | `1` | GPUs for the job; vLLM splits the model across them | | `max_tokens` | `1024` | token cap on each answer | | `temperature` | `0.0` | sampling temperature; 0 always picks the likeliest token | | `max_seq_len` | the model's own | vLLM's context length | | `chat_template_kwargs` | none | passed to the chat template, for example `{"enable_thinking": False}` | | `timeout` | none | seconds to wait for the job; none waits until it ends | `experiment`, `mlflow`, `cpu_per_worker`, `memory_per_worker`, `target_cluster`, `image`, `home_volume`, `env` and `name` work as they do for [`pm.finetune`](https://docs.privatemind.com/sdk-finetune.html#resources). Keep the eval file apart from the training data. A model scored on rows it trained on looks better than it is. To see what a fine-tune changed, run the same eval on the model you started from, with `model=` or its registry version. ## Serve it A model server serves a published version like any other registry version: ```python server = pm.launch_model_server( "gpu-cluster", gpus=1, weights="support-bot-dpo:1", vllm_version="v0.32.0", ) ``` The server pins the version to its digest when it launches. [Model servers](https://docs.privatemind.com/sdk-model-servers.html) covers the server's lifecycle and the vLLM versions on offer, and [Deploy a model](https://docs.privatemind.com/model-deployments.html) the same from the UI. ## Where next - [Fine-tuning](https://docs.privatemind.com/sdk-finetune.html): SFT, methods, checkpoints and resources. - [Model weights](https://docs.privatemind.com/sdk-weights.html): everything else you can do with a registry version. - [Training jobs](https://docs.privatemind.com/training-sdk.html): the `Run` handle, failure reasons and logs.