Get API key

Deploy a model

View as Markdown

Run your own OpenAI-compatible model server on your GPUs, from the catalog or your own weights.

Note

Deploying models needs a dedicated GPU allocation for your org and compute access on your account. If your org is on shared compute, or you don't have compute access, the Compute page and these features don't appear. Ask your org admin or account team.

A deployment is your own OpenAI-compatible vLLM inference server, pinned to GPUs your organization owns. It serves a model from the curated catalog or from your own weights. Once you register it, you call it through the same API as any built-in model.

This page covers the app. You can also deploy from a script with the Python SDK or the Model servers API. A script can run as a service identity of your org, so it does not depend on one person's key.

Launch a deployment

Go to Compute → Inference & Training and start a new deployment. You choose:

  • Model source, one of:
    • a catalog model, from the curated catalog available to your org;
    • From your models, a version your org pushed to its registry, up to 100 GiB;
    • a custom volume: your own weights on a storage volume you created in the Compute page, with an optional subpath to the model directory.
  • Host: which of your GPU hosts to run on. A host dedicated to your organization shows its name, vCPU and memory. Shared hardware shows its GPU model.
  • GPUs: the specific GPUs, or MIG slices, to pin the server to. GPUs already in use are shown, and a partly used GPU can be shared up to a memory cap.
  • vLLM version: from the supported list.
  • vCPU, memory and shared memory: optional. Leave them blank for the standard size. Shared memory is the size of /dev/shm and comes out of memory, so 600 GiB of 1 TiB leaves about 424 GiB for everything else. On a KV cache host it is at least 1.5 TiB: a smaller value is raised to 1.5 TiB, and a larger one is used as given. The deployment shows the size it got beside its memory. These requests count toward your personal compute quotas.
  • Writable volumes: optional. See Writable volumes.

Under Advanced, you can set common vLLM parameters (max-model-len, gpu-memory-utilization, tensor-parallel-size, dtype, and so on) and add any other vLLM flag as a custom key/value pair.

Note

Catalog models run from weights already staged on your hardware. The platform does not download weights from the internet, so a custom model must be pushed to your registry or placed on a volume you control before you deploy it. A registry version is pinned to its digest at launch, so a later push never changes what a restart serves.

Writable volumes

A writable volume is a volume of yours mounted read-write at a path you give, so what the server writes there survives a restart. Use one for caches. A deployment builds its compile and autotune caches again each time it starts, which on a large model can take longer than loading the weights. Mount a volume where the server writes them and the next start reads them back.

  1. Create a volume for the cache in the Volumes section, on the deployment's cluster. On a dedicated host, Local NVMe is the fastest tier.
  2. When you launch, add it under Writable volumes and set its path to the folder your image writes its caches to. vLLM keeps its own in /root/.cache/vllm, in $XDG_CACHE_HOME/vllm if your image sets XDG_CACHE_HOME, or in $VLLM_CACHE_ROOT if it sets that.

The rules:

  • You can add up to 8 volumes.
  • Each must be your org's, on the deployment's cluster, and listed once. Another member's private volume is refused.
  • A Local NVMe volume must be on the deployment's host.
  • The volume the weights come from stays read-only and can't be one of them, and neither can a workspace's home.
  • Each path is absolute and can't be on, inside or above /model, /custom, /dev, /kvcache, /proc, /sys or /root/.cache/huggingface.
  • The volumes can't be changed after launch. To change them, drop the deployment and launch it again.

Give a cache a volume of its own. A server on a platform image writes as root, which a shared volume records as an anonymous user. A workspace that mounts the same volume can read what the server writes, but can't change or delete the files in folders the server made. The server can't write into folders a workspace created unless they are writable by everyone. To clear a cache, drop the deployment, delete the volume, and launch again with a new one.

Make it callable

A deployment is not on the API until you register it. A new deployment starts up first and shows its status. If it can't be placed or its container crashes, the status shows the reason or the end of its output. Startup takes minutes for a small model and can take most of an hour for a very large model with a large KV cache. Once it's running:

  • Register the deployment and give it a public model name. That name becomes a model id you can pass to the chat completions API and any OpenAI-compatible client, exactly like a built-in model.
  • Rename changes the public model name. Unregister removes it from the API without deleting the server.
  • Drop stops the server, releases the GPUs, and unregisters it.

No translation

Tick No translation and the platform sends these routes to your vLLM as you sent them:

  • POST /v1/chat/completions
  • POST /v1/messages
  • POST /v1/messages/count_tokens
  • POST /v1/responses

Claude Code speaks the Messages API and Codex speaks the Responses API. By default, the platform turns a Messages or Responses request into a chat completion for your server and turns the answer back. It also adjusts chat requests on the way.

With No translation on, the platform sets model to your server's own name and removes priority. On a streamed chat request it also sets stream_options.include_usage to true, keeping any other stream_options you sent, so the tokens can be counted. The rest of the request reaches vLLM unchanged, and vLLM's answer comes back unchanged.

What still happens:

  • Keys, budgets, and rate and capacity limits apply as on any request.
  • Usage is recorded from the usage numbers your vLLM returns.
  • A chat request's metadata is still recorded as usage tags and passed on as sent.

What the platform skips:

  • It doesn't map reasoning_effort, thinking, output_config.effort or reasoning.effort to a reasoning level.
  • It doesn't run the moderation check.
  • It doesn't load file sources from Files.

Your vLLM gets those fields as sent. What it does with them, and with tools and thinking, depends on its own support for each route. Your vLLM must serve these routes, or the call fails with its error.

Set it when you launch, with the No translation box on the launch form or nativeApi in the Model servers API. The setting takes effect when you register the server, so a script that relaunches a server and registers it under the same name keeps it. You can also tick it when you register, or later under Edit API names. Either way the server remembers your choice for its next registration. The switch takes effect within a minute, with no restart.

Call your model

Once registered, your model is another id in the catalog:

Python
from openai import OpenAI

client = OpenAI(base_url="https://api.privatemind.com/v1", api_key="PMIND...:abcdef...")

resp = client.chat.completions.create(
    model="<your-public-model-name>",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

On a dedicated host

Note

These options exist only on a host dedicated to your organization, and only where they have been enabled for it. Ask your account team.

Local NVMe

A volume on the Local NVMe tier sits on a disk inside your host. A deployment on that host can serve weights from it as a custom volume, or keep its writable volumes on it. The tier has no replica or backup, so keep anything you can't recreate on another tier.

KV cache

A KV cache keeps the computed prefix of a prompt in host memory and on NVMe after it leaves the GPU. When a long prefix repeats, the server reloads it, which is faster than computing it again.

A deployment on a KV cache host already has the disk at /kvcache and the memory permissions the cache needs. To turn it on, add a custom key/value pair under Advanced with the key kv-transfer-config and a value like this one, written for vLLM 0.32:

JSON
{"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":1099511627776,"eviction_policy":"lru","secondary_tiers":[{"type":"fs","root_dir":"/kvcache/my-model"}]}}
  • cpu_bytes_to_use sizes the memory tier, in bytes. It counts against the deployment's memory, so set the memory well above it. Keep it above your longest prompt's KV, because a prompt that doesn't fit may not be stored. The tier lives in /dev/shm, so for a tier above 1.5 TiB, set shared memory above it too.
  • root_dir is the deployment's own folder under /kvcache. vLLM creates it. Give each deployment its own.

The platform removes the oldest files when the disk fills, and a removed block is computed again. A large memory tier adds to startup time, because it is pinned before the server answers.

Where next