Get API key

Model servers

View as Markdown

Launch and manage model servers on your GPUs from a script.

Note

These calls need a personal API key, or the token of a service identity with May manage model servers ticked. Inference only, Inference and containers, management and org application keys are refused. Launching also needs compute and GPU access on the account. Install the SDK with pip install -U privatemind: version 0.8.1 or later, or 0.9.0 or later to launch from your registry with weights, or 0.11.0 or later for LoRA adapters and volumes.

A model server is an OpenAI-compatible vLLM server pinned to GPUs your org owns. From a script, you can do most of what the Deploy a model page does in the app.

Launch a server

Python
import privatemind as pm

server = pm.launch_model_server(
    "gpu-cluster",
    gpus=8,
    volume="models-ab1cd",
    volume_subpath="my-model",
    vllm_version="v0.32.0",
    cpu="64",
    memory="512Gi",
    args={"tensor-parallel-size": "8"},
)
server.wait()
server.register(["my-model"])

Name exactly one source: a catalog model, your own volume, or a version from your org's registry as weights. args are extra vLLM flags without the leading dashes, including kv-transfer-config for a KV cache.

Argument Default Notes
target_cluster required The cluster to run on.
gpus required How many GPUs.
model none A catalog slug.
volume none The full name of a volume holding your own weights, from pm.model_server_volumes().
image none Your own image, which needs gpu_host and gpu_indices.
volume_subpath volume root The model's folder inside the volume.
weights none A <name>:<version> from your org's registry, pinned to its digest at launch. Up to 100 GiB; for a bigger version, pull it into a volume and launch from the volume.
vllm_version required One of the versions in the catalog.
args none Extra vLLM flags.
adapters none LoRA adapters to serve on the model, as {name: folder}.
volumes none More of your volumes to mount read-write, as {full name: mount path}. See Writable volumes.
gpu_host, gpu_indices free cards Name both to pin the server.
cpu 4 Total vCPU, not per GPU.
memory set by the platform Host memory, such as 512Gi.
display_name none The name shown in the app, and the start of the server's name.

See what you can run

Python
catalog = pm.model_server_catalog()
print(catalog.versions)
print([model["slug"] for model in catalog.models])

for host in pm.model_server_allocations():
    print(host["host"], host["cluster"], host["gpu_indices"])

for volume in pm.model_server_volumes():
    print(volume["metadata"]["name"])

for image in pm.model_server_images():
    print(image["image"], image["size_bytes"])

Manage a server

Call What it does
pm.list_model_servers() Your org's servers.
pm.get_model_server(name) One server by name.
server.ready, server.reason, server.message Its state. After a crash, message holds the end of its output.
server.wait(timeout=...) Waits until it answers. The default is 90 minutes. It raises TimeoutError when that runs out, and ModelServerFailedError if the server can't start, such as a crash or a volume name that doesn't exist.
server.register(aliases) Makes it callable through the API under those names, with its LoRA adapters.
server.adapters Each LoRA adapter's name and the model name that calls it.
server.unregister() Takes it off the API without stopping it.
server.delete() Stops it and releases its GPUs.

LoRA adapters

Serve several fine-tunes of one model on one server. Name each adapter with its folder, the one holding adapter_config.json:

Python
server = pm.launch_model_server(
    "gpu-cluster",
    gpus=1,
    volume="home-ab1cd",
    volume_subpath="base/qwen3-8b",
    vllm_version="v0.32.0",
    memory="40Gi",
    adapters={"statements-v1": "adapters/statements-v1", "notes-v2": "adapters/notes-v2"},
)
server.wait()
server.register(["qwen3-8b-base"])
print(server.adapters)

Registering makes each adapter callable by its name, here statements-v1 and notes-v2, as well as by the full name in server.adapters. server.register(["qwen3-8b-base"], adapters={"statements-v1": ["statements"]}) gives an adapter other names. Each register() sets every name again: pass the server's own names too, or they are cleared, and an adapter left out goes back to its own name.

A relative folder is read from volume. To serve adapters on a catalog model, mount the volume that holds them with volumes={"home-ab1cd": "/adapters"} and give absolute folders under it, such as "/adapters/statements-v1". Each adapter must be trained on the server's model, and one with a rank above 16 needs args={"max-lora-rank": "<rank>"}. The Model servers API has the rest of the rules.

As a service identity

To run as a service identity, exchange its client id and secret for a token over HTTP; the SDK does not do this for you. Then pass the token as token= to pm.reconfigure() or Client, or set it in PRIVATEMIND_TOKEN.

The token lasts 15 minutes and the SDK does not renew it, so a server.wait() that runs past that raises AuthError. Exchange again, call pm.reconfigure(token=...), and get the server again with pm.get_model_server(name).

HTTP API

The SDK calls the Model servers API.

Where next