# Model servers Launch and manage model servers on your GPUs from a script. > [!NOTE] > These calls need a personal API key, or the token of a [service identity](https://docs.privatemind.com/sdk-model-servers.html#as-a-service-identity) with **May manage model servers** ticked. `Inference only`, `Inference and containers`, management and org application keys are refused. Launching also needs compute and GPU access on the account. Install the SDK with `pip install -U privatemind`: version 0.8.1 or later, or 0.9.0 or later to launch from your [registry](https://docs.privatemind.com/registry.html) with `weights`, or 0.11.0 or later for [LoRA adapters](https://docs.privatemind.com/sdk-model-servers.html#lora-adapters) and `volumes`. A model server is an OpenAI-compatible vLLM server pinned to GPUs your org owns. From a script, you can do most of what the [Deploy a model](https://docs.privatemind.com/model-deployments.html) page does in the app. ## Launch a server ```python import privatemind as pm server = pm.launch_model_server( "gpu-cluster", gpus=8, volume="models-ab1cd", volume_subpath="my-model", vllm_version="v0.32.0", cpu="64", memory="512Gi", args={"tensor-parallel-size": "8"}, ) server.wait() server.register(["my-model"]) ``` Name exactly one source: a catalog `model`, your own `volume`, or a version from your org's registry as `weights`. `args` are extra vLLM flags without the leading dashes, including `kv-transfer-config` for a [KV cache](https://docs.privatemind.com/model-deployments.html#kv-cache). | Argument | Default | Notes | |---|---|---| | `target_cluster` | required | The cluster to run on. | | `gpus` | required | How many GPUs. | | `model` | none | A catalog slug. | | `volume` | none | The full name of a volume holding your own weights, from `pm.model_server_volumes()`. | | `image` | none | [Your own image](https://docs.privatemind.com/model-servers-api.html#run-your-own-image), which needs `gpu_host` and `gpu_indices`. | | `volume_subpath` | volume root | The model's folder inside the volume. | | `weights` | none | A `:` from your org's [registry](https://docs.privatemind.com/registry.html), pinned to its digest at launch. Up to 100 GiB; for a bigger version, [pull it](https://docs.privatemind.com/sdk-weights.html#pull) into a volume and launch from the volume. | | `vllm_version` | required | One of the versions in the catalog. | | `args` | none | Extra vLLM flags. | | `adapters` | none | [LoRA adapters](https://docs.privatemind.com/sdk-model-servers.html#lora-adapters) to serve on the model, as `{name: folder}`. | | `volumes` | none | More of your volumes to mount read-write, as `{full name: mount path}`. See [Writable volumes](https://docs.privatemind.com/model-deployments.html#writable-volumes). | | `gpu_host`, `gpu_indices` | free cards | Name both to pin the server. | | `cpu` | 4 | Total vCPU, not per GPU. | | `memory` | set by the platform | Host memory, such as `512Gi`. | | `display_name` | none | The name shown in the app, and the start of the server's name. | ## See what you can run ```python catalog = pm.model_server_catalog() print(catalog.versions) print([model["slug"] for model in catalog.models]) for host in pm.model_server_allocations(): print(host["host"], host["cluster"], host["gpu_indices"]) for volume in pm.model_server_volumes(): print(volume["metadata"]["name"]) for image in pm.model_server_images(): print(image["image"], image["size_bytes"]) ``` ## Manage a server | Call | What it does | |---|---| | `pm.list_model_servers()` | Your org's servers. | | `pm.get_model_server(name)` | One server by name. | | `server.ready`, `server.reason`, `server.message` | Its state. After a crash, `message` holds the end of its output. | | `server.wait(timeout=...)` | Waits until it answers. The default is 90 minutes. It raises `TimeoutError` when that runs out, and `ModelServerFailedError` if the server can't start, such as a crash or a volume name that doesn't exist. | | `server.register(aliases)` | Makes it callable through the [API](https://docs.privatemind.com/chat-completions.html) under those names, with its LoRA adapters. | | `server.adapters` | Each LoRA adapter's name and the model name that calls it. | | `server.unregister()` | Takes it off the API without stopping it. | | `server.delete()` | Stops it and releases its GPUs. | ## LoRA adapters Serve several fine-tunes of one model on one server. Name each adapter with its folder, the one holding `adapter_config.json`: ```python server = pm.launch_model_server( "gpu-cluster", gpus=1, volume="home-ab1cd", volume_subpath="base/qwen3-8b", vllm_version="v0.32.0", memory="40Gi", adapters={"statements-v1": "adapters/statements-v1", "notes-v2": "adapters/notes-v2"}, ) server.wait() server.register(["qwen3-8b-base"]) print(server.adapters) ``` Registering makes each adapter callable by its name, here `statements-v1` and `notes-v2`, as well as by the full name in `server.adapters`. `server.register(["qwen3-8b-base"], adapters={"statements-v1": ["statements"]})` gives an adapter other names. Each `register()` sets every name again: pass the server's own names too, or they are cleared, and an adapter left out goes back to its own name. A relative folder is read from `volume`. To serve adapters on a catalog `model`, mount the volume that holds them with `volumes={"home-ab1cd": "/adapters"}` and give absolute folders under it, such as `"/adapters/statements-v1"`. Each adapter must be trained on the server's model, and one with a rank above 16 needs `args={"max-lora-rank": ""}`. The [Model servers API](https://docs.privatemind.com/model-servers-api.html#lora-adapters) has the rest of the rules. ## As a service identity To run as a service identity, [exchange its client id and secret for a token over HTTP](https://docs.privatemind.com/model-servers-api.html#service-identities); the SDK does not do this for you. Then pass the token as `token=` to `pm.reconfigure()` or [`Client`](https://docs.privatemind.com/sdk.html#authentication), or set it in `PRIVATEMIND_TOKEN`. The token lasts 15 minutes and the SDK does not renew it, so a `server.wait()` that runs past that raises `AuthError`. Exchange again, call `pm.reconfigure(token=...)`, and get the server again with `pm.get_model_server(name)`. ## HTTP API The SDK calls the [Model servers API](https://docs.privatemind.com/model-servers-api.html). ## Where next - [Deploy a model](https://docs.privatemind.com/model-deployments.html): the same thing in the app, and the options a dedicated host adds. - [Training jobs](https://docs.privatemind.com/training-sdk.html): run training from a script.