Get API key

Evaluate and publish

View as Markdown

Push a fine-tune to your org's registry, score it on your own eval set, and serve it through a model server.

A fine-tune ends with a result on your Volume: an adapter, or a whole model. Three steps take it from there to something your applications call. run.publish pushes it to your org's registry as a model version. pm.evaluate scores a model on an eval set of your own. A model server then serves the version you choose.

All three need SDK 0.11.0 or later, and publishing needs the registry turned on for your org. Run publish and evaluate in a workspace on the home Volume your fine-tunes use: both stage files there and read results back from it.

Publish a fine-tune

Python
import privatemind as pm

run = pm.get_job("ft-support-dpo")       # or the Run that pm.finetune returned
run.wait()
ref = run.publish("support-bot-dpo")
print(ref)                               # support-bot-dpo:1

publish(name, version=None) takes a model name with the registry's naming rules and pushes the result as a version of that model. It returns the reference, <name>:<version>. Leave version out and it is one more than the model's highest numeric version, so 1 for a new model.

For a LoRA or QLoRA run, publish first merges the adapter into the base model, in a one-GPU job on the same cluster, and waits for it. The version is then a complete model that needs nothing else to load. A full-parameter run is pushed as trained. publish takes a run that succeeded, which pm.finetune submitted from this home Volume in the last 14 days, and raises ValidationError for any other.

The new version is an ordinary registry version: list it, pull it, share it or delete it like any other.

Evaluate a model

pm.evaluate answers every prompt in an eval file with the model and scores each answer with a grader:

Python
result = pm.evaluate(
    weights="support-bot-dpo:1",
    dataset="data/eval.jsonl",
    grader="string_check",
)
print(result.mean, len(result.scores))

It runs as a training job on your GPU allocation, on the training-rl image, and waits for it. vLLM writes one answer to each prompt, and the grader scores each answer. evaluate returns an EvalResult: scores holds one score per row, in the file's order, mean their mean, and run the job. If the job doesn't succeed, evaluate raises PrivatemindError with its failure message. The mean and the row count also go to MLflow when your org has it set up.

The eval file has the same shape as a GRPO dataset: one prompt per line as messages, plus any fields the grader needs. The grader works as a GRPO reward does: "string_check" against the row's answer field, or a grade(sample, item) -> float function of your own. So the same function can train a model with GRPO and then measure it. A <think>...</think> block is cut from each answer before it is graded.

Argument Default What it does
weights or model required, one of them a <name>:<version> from your registry, or a Hugging Face id or Volume path
dataset required the eval file, under your home Volume
grader required "string_check", or a grade(sample, item) function
reference_field "answer" the field string_check compares with
gpus 1 GPUs for the job; vLLM splits the model across them
max_tokens 1024 token cap on each answer
temperature 0.0 sampling temperature; 0 always picks the likeliest token
max_seq_len the model's own vLLM's context length
chat_template_kwargs none passed to the chat template, for example {"enable_thinking": False}
timeout none seconds to wait for the job; none waits until it ends

experiment, mlflow, cpu_per_worker, memory_per_worker, target_cluster, image, home_volume, env and name work as they do for pm.finetune.

Keep the eval file apart from the training data. A model scored on rows it trained on looks better than it is. To see what a fine-tune changed, run the same eval on the model you started from, with model= or its registry version.

Serve it

A model server serves a published version like any other registry version:

Python
server = pm.launch_model_server(
    "gpu-cluster",
    gpus=1,
    weights="support-bot-dpo:1",
    vllm_version="v0.32.0",
)

The server pins the version to its digest when it launches. Model servers covers the server's lifecycle and the vLLM versions on offer, and Deploy a model the same from the UI.

Where next