# Model servers Launch, manage and register model servers over HTTP. A model server is an OpenAI-compatible vLLM server pinned to GPUs your org owns. This API does what the [Deploy a model](https://docs.privatemind.com/model-deployments.html) page does in the app, for scripts and CI. The same calls are in the [Python SDK](https://docs.privatemind.com/sdk-model-servers.html), and the machine-readable spec is [`/openapi-model-servers.yaml`](https://docs.privatemind.com/openapi-model-servers.yaml). All paths below are on `https://privatemind.com/api/v1/model-servers`. Once a server is registered, you call it through `api.privatemind.com` like any other model. ## Authentication Send a personal API key in the `Authorization` header, access key id and secret joined by a colon: ```text Authorization: Bearer : ``` `Inference only`, `Inference and containers` and management keys get `403`, and so does an org application key. A read-only key can only read. Launching also needs compute access and GPU access on the account. ### Service identities For a script that should not depend on one person's key, an organization admin creates a [service identity](https://docs.privatemind.com/remote-hosts.html#unattended-access-for-ci-service-identities) with **May manage model servers** ticked (**Settings → Organization → Service Identities**). The identity is a member of your org in its own right: it has its own compute quota and GPU access, the servers it launches show its name as the launcher, and every launch, delete, register and unregister it makes lands in your audit log. Service identities are switched on per org by PrivateMind; if the panel says they are not enabled yet, ask us. Exchange the identity's client id and secret for a token, then send the token as the bearer. The secret is read from the environment, so it never appears in curl's arguments: ```bash export PM_CLIENT_ID=pmsvc_… export PM_CLIENT_SECRET=pmsec_… TOKEN=$(jq -n '{client_id: env.PM_CLIENT_ID, client_secret: env.PM_CLIENT_SECRET}' \ | curl -s "https://privatemind.com/api/auth/service/token" \ -H "Content-Type: application/json" -d @- \ | jq -r .access_token) curl -s "https://privatemind.com/api/v1/model-servers" \ -H "Authorization: Bearer $TOKEN" ``` A token lasts 15 minutes (`expires_in`). A script that waits for a server to come up before calling `gateway-register` should exchange for a fresh token first. An identity without the model servers grant gets `403`. A disabled identity, a rotated secret, or an org with service identities switched off gets `401` on the exchange and on every call. If an identity you want to retire still owns running servers, disable it. Deleting it leaves the servers running with no launcher shown. ## See what you can run ```bash curl -s "https://privatemind.com/api/v1/model-servers/catalog" \ -H "Authorization: Bearer $PMIND_USER_KEY" ``` `body` holds the catalog models, each with its `slug`, `modelType` and `compatibleVllmVersions`. `total.supportedVersions` holds the vLLM versions on offer. ```bash curl -s "https://privatemind.com/api/v1/model-servers/allocations" \ -H "Authorization: Bearer $PMIND_USER_KEY" ``` `total.hosts` holds your org's GPU hosts. You can launch on a host where `deployable` is `true`. `claims` shows what already holds each card. On a host dedicated to your org, `dedicated` is `true` and you also get its name, `allocatable_cpu` and `allocatable_memory_gib`. On shared hardware `host` is an opaque ID such as `host-3f9a1c2e`, which you can pass back as `gpuHost`: ```json { "success": true, "total": { "org": "your-org", "hosts": [ { "host": "gpu-host-1", "cluster": "gpu-cluster", "deployable": true, "total_gpus": 8, "dedicated": true, "allocatable_cpu": 344, "allocatable_memory_gib": 2011, "gpu_indices": [0, 1, 2, 3, 4, 5, 6, 7], "claims": {} } ] } } ``` ```bash curl -s "https://privatemind.com/api/v1/model-servers/volumes" \ -H "Authorization: Bearer $PMIND_USER_KEY" ``` `body` holds the volumes you can see. Each one's `metadata.name` is the full name a launch needs, such as `models-ab1cd`. ## Launch a server ```bash curl -s -X POST "https://privatemind.com/api/v1/model-servers" \ -H "Authorization: Bearer $PMIND_USER_KEY" \ -H "Content-Type: application/json" \ -d '{ "targetCluster": "gpu-cluster", "gpuCount": 8, "vllmVersion": "v0.32.0", "weightsVolume": "models-ab1cd", "weightsSubPath": "my-model", "displayName": "my-model", "cpu": "64", "memory": "512Gi", "args": {"tensor-parallel-size": "8"} }' ``` | Field | Required | Notes | |---|---|---| | `targetCluster` | yes | The cluster to run on. | | `gpuCount` | yes | How many GPUs, up to 8, or MIG slices in `Shared` mode. | | `vllmVersion` | yes | One of `supportedVersions`. | | `modelSlug` | one of these three | A catalog model. Its `compatibleVllmVersions` must include the version. | | `weightsVolume` | one of these three | The full name of a volume holding your own weights. A name that doesn't exist is accepted, and the server then stops at `WeightsUnresolved`. | | `weights` | one of these three | A `:` from your org's [registry](https://docs.privatemind.com/registry.html), pinned to its digest at launch. A version that doesn't exist or is over 100 GiB is refused with `400`, and one private to another member with `403`. | | `weightsSubPath` | no | The model's folder inside the volume. Defaults to the volume root. | | `gpuHost`, `gpuIndices` | no | Name both to pin the server, with one index per GPU. Leave both out and free cards are picked for you. | | `gpuSharingMode` | no | `Exclusive`, the default, or `Shared` for a MIG slice. `Shared` needs `gpuHost` and exactly one index in `gpuIndices`. | | `gpuMemoryUtilization` | no | vLLM's `--gpu-memory-utilization`, as a string. | | `cpu` | no | Total vCPU, not per GPU, as a string. Defaults to 4. | | `memory` | no | Host memory as a string, a whole number in `Mi`, `Gi` or `Ti`, such as `512Gi`. | | `sharedMemory` | no | The size of `/dev/shm`, in the same units, such as `600Gi`. It comes out of `memory`, so it can't be larger. Blank keeps the standard size. On a KV cache host it is at least 1.5 TiB, and a larger value raises it. | | `args` | no | Extra vLLM flags without the leading dashes, all values as strings. This is how you set a [KV cache](https://docs.privatemind.com/model-deployments.html#kv-cache). | | `displayName` | no | The name shown in the app, and the start of the server's name. | | `nativeApi` | no | `true` to launch with [No translation](https://docs.privatemind.com/model-deployments.html#no-translation): registering the server turns it on. Left out, it is off. | | `image` | no | [Your own image](https://docs.privatemind.com/model-servers-api.html#run-your-own-image), which needs `gpuHost` and `gpuIndices`. | | `adapters` | no | Up to 32 [LoRA adapters](https://docs.privatemind.com/model-servers-api.html#lora-adapters) to serve on the model, as a list of `{"name": "", "path": ""}`. | | `volumes` | no | Up to 8 of your volumes to mount read-write, as a list of `{"volume": "", "mountPath": "/absolute/path"}`, such as a cache that should survive a restart. They are fixed at launch. See [Writable volumes](https://docs.privatemind.com/model-deployments.html#writable-volumes) for the rules; a launch that breaks one is refused with the reason. | The response is `201`, with the server's generated `name`: ```json { "success": true, "total": { "namespace": "...", "name": "my-model-ab1cd", "spec": { "targetCluster": "gpu-cluster", "gpuCount": 8, "gpuHost": "gpu-host-1" } } } ``` ## Read a server ```bash curl -s "https://privatemind.com/api/v1/model-servers/my-model-ab1cd" \ -H "Authorization: Bearer $PMIND_USER_KEY" ``` `total.cr` is the full server object. Read its state from the condition of type `Ready`. A server just launched may have no `status` yet. ```json { "success": true, "total": { "namespace": "...", "name": "my-model-ab1cd", "cr": { "spec": { "gpuHost": "gpu-host-1", "gpuIndices": [0, 1, 2, 3, 4, 5, 6, 7] }, "status": { "conditions": [ { "type": "Ready", "status": "True", "reason": "Reconciled", "message": "vLLM is serving." } ] } } } } ``` | `reason` | Meaning | |---|---| | `WorkloadStarting` | Still starting. | | `Reconciled` | Serving, with `status` `True`. | | `ContainerCrashing` | The container crashed. `message` holds the end of its output. | | `WeightsUnresolved` | The weights can't be reached. `message` says why. A volume name that doesn't exist stays here, so delete the server and launch again. | | `VolumesUnresolved` | A writable volume can't be mounted. `message` says why. A Local NVMe volume on another host stays here, so delete the server and launch again. | | `Unschedulable` | It can't be placed yet. `message` says why. | | `FailedCreate` | The cluster refused the server. `message` says why. | Other reasons can appear: read `message`. ## List servers ```bash curl -s "https://privatemind.com/api/v1/model-servers" \ -H "Authorization: Bearer $PMIND_USER_KEY" ``` `body` holds one server object per server, with the same shape as `total.cr`, plus its API state: `registered`, `gatewayApiModelName`, `gatewayAliases`, `gatewayNativeApi`, and `adapters`, one `{"name", "model", "registered", "aliases"}` per [LoRA adapter](https://docs.privatemind.com/model-servers-api.html#lora-adapters). ## Register a server A server isn't on the API until you register it. It has to be serving first: registering earlier answers `409`. ```bash curl -s -X POST "https://privatemind.com/api/v1/model-servers/my-model-ab1cd/gateway-register" \ -H "Authorization: Bearer $PMIND_USER_KEY" \ -H "Content-Type: application/json" \ -d '{"aliases": ["my-model"]}' ``` The model id is the server's `name`. `aliases` adds up to three names to call it by. Each is 1 to 63 letters, digits, dots, underscores or dashes, and starts and ends with a letter or digit. A name already taken answers `400`. Registering again replaces the aliases, so a call with none clears them. A chat model's name then works on the [chat completions API](https://docs.privatemind.com/chat-completions.html). `nativeApi`, `true` or `false`, turns [No translation](https://docs.privatemind.com/model-deployments.html#no-translation) on or off. If you leave it out, a new registration takes the setting the server was launched with, which is off unless the launch sent `nativeApi: true`. A registered server keeps the setting it has. A value you send also becomes the server's own setting, so a later registration keeps it. To flip it on a registered server, register again with the other value and send its `aliases` again in the same call: ```bash curl -s -X POST "https://privatemind.com/api/v1/model-servers/my-model-ab1cd/gateway-register" \ -H "Authorization: Bearer $PMIND_USER_KEY" \ -H "Content-Type: application/json" \ -d '{"aliases": ["my-model"], "nativeApi": true}' ``` ```bash curl -s -X POST "https://privatemind.com/api/v1/model-servers/my-model-ab1cd/gateway-unregister" \ -H "Authorization: Bearer $PMIND_USER_KEY" ``` Unregistering takes the server off the API and leaves it running. It takes the server's LoRA adapters off too. ## LoRA adapters A server can serve LoRA adapters on its model, each callable as its own model. Name each adapter and the folder that holds it: ```bash curl -s -X POST "https://privatemind.com/api/v1/model-servers" \ -H "Authorization: Bearer $PMIND_USER_KEY" \ -H "Content-Type: application/json" \ -d '{ "targetCluster": "gpu-cluster", "gpuCount": 1, "vllmVersion": "v0.32.0", "weightsVolume": "home-ab1cd", "weightsSubPath": "base/qwen3-8b", "memory": "40Gi", "adapters": [ {"name": "statements-v1", "path": "adapters/statements-v1"}, {"name": "notes-v2", "path": "adapters/notes-v2"} ] }' ``` - `name` is 1 to 40 lowercase letters, digits and dashes, starting and ending with a letter or digit. - `path` is the adapter's folder, the one holding `adapter_config.json`. A relative path is read from `weightsVolume`, or from the `weights` version. An absolute path must sit inside one of the server's `volumes`, which is how you serve adapters on a catalog model. - Each adapter must be trained on the server's model. An adapter with a rank above 16 needs `"args": {"max-lora-rank": ""}`, a value from 1, 8, 16, 32, 64, 128, 256, 320 or 512. - Up to four adapters run in the same batch. `"args": {"max-loras": ""}` raises that, at the cost of GPU memory. - `args` cannot carry `lora-modules` itself, and a server with adapters cannot be pooled. Registering the server registers each adapter as its own model, named `-` and callable by its adapter name too. To call an adapter by other names, give them per adapter in the same call; an empty list leaves only the full name: ```bash curl -s -X POST "https://privatemind.com/api/v1/model-servers/my-model-ab1cd/gateway-register" \ -H "Authorization: Bearer $PMIND_USER_KEY" \ -H "Content-Type: application/json" \ -d '{"aliases": ["qwen3-8b-base"], "adapters": {"statements-v1": ["statements"], "notes-v2": []}}' ``` The response lists each adapter with its `model` name and `aliases`. The same naming rules apply as for `aliases`. Registering again moves the names, and an adapter left out of the call goes back to its own name. An adapter name the server does not serve answers `400`. ## Delete a server ```bash curl -s -X DELETE "https://privatemind.com/api/v1/model-servers/my-model-ab1cd" \ -H "Authorization: Bearer $PMIND_USER_KEY" ``` This stops the server, releases its GPUs and takes it off the API. ```json { "success": true, "total": { "namespace": "...", "name": "my-model-ab1cd", "deleted": true } } ``` ## Run your own image You can run your own vLLM image on a host your org owns. PrivateMind switches custom images on per org, so ask your PrivateMind contact to start. Build from a vLLM image, so that `/bin/sh` and `vllm` are there and the image starts the server from the arguments it's given: ```dockerfile FROM vllm/vllm-openai:v0.32.0 COPY my-vllm/ /opt/my-vllm/ RUN pip install --no-deps /opt/my-vllm ``` Log in with your API key and push under your org's path: ```bash docker login privatemind.com -u "$ACCESS_KEY_ID" -p "$SECRET_ACCESS_KEY" docker build -t privatemind.com/customers/your-org/vllm:my-build . docker push privatemind.com/customers/your-org/vllm:my-build ``` List what you've pushed and how much of your storage it uses: ```bash curl -s "https://privatemind.com/api/v1/model-servers/images" \ -H "Authorization: Bearer $PMIND_USER_KEY" ``` ```json { "success": true, "body": [ { "image": "customers/your-org/vllm:my-build", "digest": "sha256:...", "size_bytes": 10737418240, "created_at": "2026-10-30T12:00:00Z" } ], "total": { "path": "customers/your-org", "used_bytes": 10737418240, "quota_bytes": 214748364800 } } ``` Launch it by adding `image`, pinned to a host your org alone holds: ```bash curl -s -X POST "https://privatemind.com/api/v1/model-servers" \ -H "Authorization: Bearer $PMIND_USER_KEY" \ -H "Content-Type: application/json" \ -d '{ "targetCluster": "gpu-cluster", "gpuHost": "gpu-host-1", "gpuIndices": [0, 1, 2, 3], "gpuCount": 4, "vllmVersion": "v0.32.0", "image": "customers/your-org/vllm:my-build", "weightsVolume": "models-ab1cd", "displayName": "my-model" }' ``` `vllmVersion` still sets which flags `args` accepts. The tag is resolved to a digest at launch, so pushing that tag again doesn't change a running server. Registering, KV cache and monitoring work as they do on any model server. Your org has a storage limit. Once it's full, a new push answers `denied`, and deleting a tag frees the space: ```bash curl -s -u "$ACCESS_KEY_ID:$SECRET_ACCESS_KEY" -X DELETE \ "https://privatemind.com/v2/customers/your-org/vllm/manifests/my-build" ``` Give each build its own tag. We keep a repository's 30 most recently pushed tags, plus any tag pushed in the last 30 days. An image whose tag was pushed over is removed, and a server still using it can't restart. ## Errors A refused call has `"success": false` and a reason in `error` or `message`. `error` is usually a string and can be an object with a `type` and `message`. | Status | Cause | |---|---| | `400` | A bad field, such as an unsupported `vllmVersion`, an unknown `modelSlug`, `gpuHost` without `gpuIndices`, an alias already taken, or a `nativeApi` that isn't `true` or `false`. | | `401` | A key that isn't valid. No key, on any call but a launch. A service identity token that has expired, or whose identity is disabled, whose secret was rotated, or whose org has service identities switched off. | | `402` | The key's budget is spent. | | `403` | `error` says which of these it is: no key on a launch; no compute or GPU access on the account; a key type this API refuses; a service identity without the model servers grant; a volume you can't use; a request over a personal limit or the org's budget on that cluster; a card the org doesn't own, or one already held beyond the shared memory cap; or a value the cluster refuses, such as an `args` value or a `Shared` request on a card without MIG. | | `404` | No such server in your org. | | `409` | No host with enough free GPUs and memory for an unpinned launch, or a server that isn't serving yet when you register it. | | `422` | A malformed value, such as `memory` without a unit the API takes. | | `5xx` | Something failed on our side. Retry reads. Check before retrying a launch. |