These calls need a personal API key, or the token of a service identity with May manage model servers ticked. Inference only, Inference and containers, management and org application keys are refused. Launching also needs compute and GPU access on the account. Install the SDK with pip install -U privatemind: version 0.8.1 or later, or 0.9.0 or later to launch from your registry with weights, or 0.11.0 or later for LoRA adapters and volumes.
A model server is an OpenAI-compatible vLLM server pinned to GPUs your org owns. From a script, you can do most of what the Deploy a model page does in the app.
Launch a server
import privatemind as pm
server = pm.launch_model_server(
"gpu-cluster",
gpus=8,
volume="models-ab1cd",
volume_subpath="my-model",
vllm_version="v0.32.0",
cpu="64",
memory="512Gi",
args={"tensor-parallel-size": "8"},
)
server.wait()
server.register(["my-model"])Name exactly one source: a catalog model, your own volume, or a version from your org's registry as weights. args are extra vLLM flags without the leading dashes, including kv-transfer-config for a KV cache.
| Argument | Default | Notes |
|---|---|---|
target_cluster |
required | The cluster to run on. |
gpus |
required | How many GPUs. |
model |
none | A catalog slug. |
volume |
none | The full name of a volume holding your own weights, from pm.model_server_volumes(). |
image |
none | Your own image, which needs gpu_host and gpu_indices. |
volume_subpath |
volume root | The model's folder inside the volume. |
weights |
none | A <name>:<version> from your org's registry, pinned to its digest at launch. Up to 100 GiB; for a bigger version, pull it into a volume and launch from the volume. |
vllm_version |
required | One of the versions in the catalog. |
args |
none | Extra vLLM flags. |
adapters |
none | LoRA adapters to serve on the model, as {name: folder}. |
volumes |
none | More of your volumes to mount read-write, as {full name: mount path}. See Writable volumes. |
gpu_host, gpu_indices |
free cards | Name both to pin the server. |
cpu |
4 | Total vCPU, not per GPU. |
memory |
set by the platform | Host memory, such as 512Gi. |
display_name |
none | The name shown in the app, and the start of the server's name. |
See what you can run
catalog = pm.model_server_catalog()
print(catalog.versions)
print([model["slug"] for model in catalog.models])
for host in pm.model_server_allocations():
print(host["host"], host["cluster"], host["gpu_indices"])
for volume in pm.model_server_volumes():
print(volume["metadata"]["name"])
for image in pm.model_server_images():
print(image["image"], image["size_bytes"])Manage a server
| Call | What it does |
|---|---|
pm.list_model_servers() |
Your org's servers. |
pm.get_model_server(name) |
One server by name. |
server.ready, server.reason, server.message |
Its state. After a crash, message holds the end of its output. |
server.wait(timeout=...) |
Waits until it answers. The default is 90 minutes. It raises TimeoutError when that runs out, and ModelServerFailedError if the server can't start, such as a crash or a volume name that doesn't exist. |
server.register(aliases) |
Makes it callable through the API under those names, with its LoRA adapters. |
server.adapters |
Each LoRA adapter's name and the model name that calls it. |
server.unregister() |
Takes it off the API without stopping it. |
server.delete() |
Stops it and releases its GPUs. |
LoRA adapters
Serve several fine-tunes of one model on one server. Name each adapter with its folder, the one holding adapter_config.json:
server = pm.launch_model_server(
"gpu-cluster",
gpus=1,
volume="home-ab1cd",
volume_subpath="base/qwen3-8b",
vllm_version="v0.32.0",
memory="40Gi",
adapters={"statements-v1": "adapters/statements-v1", "notes-v2": "adapters/notes-v2"},
)
server.wait()
server.register(["qwen3-8b-base"])
print(server.adapters)Registering makes each adapter callable by its name, here statements-v1 and notes-v2, as well as by the full name in server.adapters. server.register(["qwen3-8b-base"], adapters={"statements-v1": ["statements"]}) gives an adapter other names. Each register() sets every name again: pass the server's own names too, or they are cleared, and an adapter left out goes back to its own name.
A relative folder is read from volume. To serve adapters on a catalog model, mount the volume that holds them with volumes={"home-ab1cd": "/adapters"} and give absolute folders under it, such as "/adapters/statements-v1". Each adapter must be trained on the server's model, and one with a rank above 16 needs args={"max-lora-rank": "<rank>"}. The Model servers API has the rest of the rules.
As a service identity
To run as a service identity, exchange its client id and secret for a token over HTTP; the SDK does not do this for you. Then pass the token as token= to pm.reconfigure() or Client, or set it in PRIVATEMIND_TOKEN.
The token lasts 15 minutes and the SDK does not renew it, so a server.wait() that runs past that raises AuthError. Exchange again, call pm.reconfigure(token=...), and get the server again with pm.get_model_server(name).
HTTP API
The SDK calls the Model servers API.
Where next
- Deploy a model: the same thing in the app, and the options a dedicated host adds.
- Training jobs: run training from a script.