The SDK uses your org's GPU allocation. If your org doesn't have one, training calls return an error. See Compute & training.
The PrivateMind Python SDK submits and manages training jobs, with the same calls from a notebook, a script or CI. It comes installed in every workspace, with your token, the API URLs and your GPU cluster name already set as environment variables. There, you import privatemind and call submit_training.
Failure reasons, logs, stopping a run, registry weights and the privatemind jobs command need SDK 0.11.0 or later: pip install -U "privatemind>=0.11".
Authentication
In a workspace you can skip this section, because credentials and URLs are already set.
Outside a workspace, on a laptop or in CI, the SDK looks for a token in this order:
token=passed toClient(...)- the
PRIVATEMIND_TOKENenvironment variable ~/.privatemind/auth(must be mode0600, owned by you, and not a symlink)
The token is a PrivateMind API key. The SDK talks to two services, and training uses the second:
- Platform API:
base_url=orPRIVATEMIND_URL, defaulthttps://api.privatemind.com. Serves image generation and batch. - App backend:
app_url=orPRIVATEMIND_APP_URL, defaulthttps://privatemind.com. Serves the training methods (submit_training,list_jobs,get_job).
Both must use https://. The SDK overview covers the rest of the configuration.
Submit a job
submit_training starts a job and returns a Run handle. All its arguments are keyword-only.
from privatemind import submit_training
run = submit_training(
entrypoint="python train.py --epochs 10",
image="<training-image>",
workers=1,
target_cluster="<your-gpu-cluster>",
gpus_per_worker=1,
)
print(run) # Run(name='tj-abc12', phase='Pending')
print(run.url) # link to the run in the PrivateMind UI
run.wait() # block until the job is terminal
print(run.phase, run.mlflow_run_id)image must be one of the platform's own training images. The platform refuses any other image, your own included. There are two: the default PyTorch image, and an image that adds vLLM, which GRPO and evaluation run on. On SDK 0.4.0 or newer, Client().list_training_images() lists every offered version of both, with the default first. Pass an entry's value unchanged, digest included.
| Parameter | Default | Purpose |
|---|---|---|
entrypoint |
required | Shell command the job runs on its Ray head, e.g. python train.py --epochs 10. |
image |
required | Container image the workers run, from the platform's list (see above). |
workers |
required | Number of Ray workers. Training is supported on one node: use 1, and set the GPUs with gpus_per_worker. The SDK refuses more than 64, or 16 for a job with GPUs. |
target_cluster |
required | Your org's GPU cluster name (from your dashboard or account team). |
gpus_per_worker |
0 |
GPUs per worker, 0 to 8. 0 is CPU-only. |
cpu_per_worker |
none | CPU request per worker: an int or a Kubernetes quantity string ("2", "500m"). |
memory_per_worker |
none | Memory request per worker, same format ("8Gi"). |
env |
none | Environment variables for the workers. Keys starting with MLFLOW_ are reserved by the platform and rejected. |
mlflow |
True |
Track the run in MLflow, which fills in run.mlflow_run_id. If your org has no MLflow set up, pass False or the job is refused. |
ray_version |
platform default | Pin the Ray version the job runs. |
ttl_seconds_after_finished |
600 |
Seconds the job's Ray cluster stays up after the job ends, whether it succeeded or failed, 0 to 86400 (one day). The platform then removes the cluster. The run's record stays until you delete it. |
name |
assigned | Job name: lowercase letters, digits and hyphens, up to 40 characters, starting and ending with a letter or digit. Omit it and the platform assigns one. |
namespace |
assigned | Advanced override. Omit it in normal use. |
volumes |
none | Volumes mounted into every worker. See Mounting data. |
weights |
none | Versions from your org's registry to mount read-only. See Mounting registry weights. |
gpu_placements |
auto-derived | Pin exact GPUs. Omit it and the platform picks them. See GPUs. |
webhook_url |
none | HTTPS URL the platform POSTs to when the job finishes. See Completion webhooks. |
webhook_secret |
none | Signs each webhook delivery. Requires webhook_url. |
client |
shared client | An explicit Client to submit through. |
The SDK checks arguments before it sends anything. A bad value, such as workers=0, a malformed name or a reserved env key, raises ValidationError.
A run's RAM and vCPU count against your personal compute allowance, which workspaces, inference deployments and jobs also use. The total covers every worker plus the job's Ray head, the node that runs your entrypoint. See Personal compute quotas.
In a workspace, os.environ["PRIVATEMIND_TARGET_CLUSTER"] holds your cluster name. The @pm.train decorator fills in target_cluster and image for you.
GPUs
Usually you set gpus_per_worker and leave gpu_placements unset. The platform then pins the job to free GPUs your org owns, so you don't need to know which ones you have.
To pin exact devices, pass one placement per worker. Each is a mapping with a host and exactly gpus_per_worker GPU indices (integers 0 to 7, no duplicates):
gpu_placements=[{"host": "<gpu-host>", "indices": [0, 1]}] # workers=1, gpus_per_worker=2The SDK checks the shape when you submit: gpu_placements needs gpus_per_worker above 0, and the list length must equal workers. The platform checks ownership, index conflicts and quota, and rejects a job that asks for GPUs it cannot have.
Your entrypoint runs on the job's Ray head, which has no GPU. The GPUs are on the workers, so your script reaches them through Ray (Ray Train, or ray.remote(num_gpus=...)) as in any Ray program. The @pm.train decorator does this for you.
The Run handle
A Run is a live handle to the job. Its name, namespace and url never change. Its status is a snapshot. Status properties such as run.phase return the last status fetched, and fetch it the first time you read one. Call run.refresh() to fetch it again.
| Member | What it gives you |
|---|---|
run.phase |
Lifecycle state: Pending, Queued or Running while the job is live, then Succeeded, Failed, Stopped or Terminating. |
run.url |
Link to the run in the PrivateMind UI. |
run.failure_reason |
Why a failed run failed, as one word, else None. See When a run fails. |
run.failure_message |
The last 8 KiB of the failure detail, else None. |
run.log_path |
Where the job writes its full log, as a path inside the job. None when the job mounts no writable Volume. |
run.logs() |
The job's driver log, or the failure message where the log can't be read. See Logs. |
run.mlflow_run_id |
MLflow run id once tracking has started, else None. |
run.ray_job_name |
Underlying Ray submission id once the job is admitted, else None. |
run.start_time, run.end_time |
RFC 3339 timestamps, None until the job starts or finishes. |
run.webhook_status |
Completion-webhook delivery state. See Checking delivery from the SDK. |
run.refresh() |
Fetch the status again. Returns the run. |
run.wait(timeout=None, poll_interval=5.0) |
Block until the job stops running. Returns the run. |
run.watch(timeout=None, poll_interval=5.0) |
Block like wait(), showing a live status table in a notebook. Returns the run. |
run.stop() |
End the job and keep its record. See Stop or delete a run. |
run.cancel() |
The same as stop(). |
run.delete() |
End the job if it still runs, and remove its record. |
run.publish(name, version=None) |
Push a fine-tune's result to your org's registry and return <name>:<version>. See Evaluate and publish. |
run.metrics() |
Latest metric values from this run's MLflow run, as {name: float}. |
run.params() |
Params logged to this run's MLflow run, as {name: str}. |
run.list_artifacts(path=None) |
Artifact entries under the run, optionally rooted at path. |
A job reports one of seven phases. The SDK groups them into two sets, both importable from privatemind:
| Phase | Set | Meaning |
|---|---|---|
Pending |
ACTIVE_PHASES |
the job was accepted; the platform has not started on it yet |
Queued |
ACTIVE_PHASES |
the workers exist and are waiting for GPUs to come free |
Running |
ACTIVE_PHASES |
the workers are up and your entrypoint is executing |
Succeeded |
TERMINAL_PHASES |
the job exited cleanly |
Failed |
TERMINAL_PHASES |
the job exited non-zero, or the platform could not run it. run.failure_reason says which |
Stopped |
TERMINAL_PHASES |
you or an admin stopped the job |
Terminating |
TERMINAL_PHASES |
tear-down is under way. You are unlikely to see it: the platform doesn't report it, and a deleted job stops answering. The SDK counts it as terminal so a job that reaches it ends the wait |
wait() polls while the phase is in ACTIVE_PHASES and returns as soon as it isn't.
poll_intervalis the first gap between polls, in seconds. It doubles each poll, up to 30 seconds, or up topoll_intervalif you set it higher.timeout=Nonewaits forever. If the job is still running when a timeout expires,wait()raisesTimeoutError. The job keeps running until it finishes or you cancel it.
A phase in neither set raises UnknownPhaseError, which carries .run_name and .phase. It means the platform reports a phase your installed SDK doesn't know. The SDK then can't tell whether the job has finished, so it won't say it has. Upgrade privatemind, or read run.phase and decide yourself. SDK versions before 0.11.0 don't know Stopped, so waiting on a stopped run raises this error there.
wait(on_unknown_phase="return") returns with a UserWarning and doesn't raise, but keep the default. If your notebook runs warnings.filterwarnings("ignore"), that return is silent. run.metrics() then gives you half a training run's numbers with nothing on screen to say so.
try:
run.wait(timeout=3600)
except TimeoutError:
run.stop()
if run.phase == "Succeeded":
print(run.mlflow_run_id, run.end_time)
elif run.phase == "Failed":
print(run.failure_reason)
print(run.logs())You can list or look up jobs without the original handle:
list_jobs()returns aRunListof handles with their status filled in: aRunfor each training run and aJobfor each workspace job. Each item'skindis"training"or"job". On SDK versions before 0.5.0 it always returns an empty list, so upgrade first.phase=keeps only jobs in that phase, for both kinds. The SDK does the filtering, on an exact match that ignores case.list_training_jobs()returns only the training runs.get_job(name)looks up a training run, andget_workspace_job(name)a workspace job.
from privatemind import list_jobs, get_job
for r in list_jobs(): # training runs and workspace jobs you may see
print(r.kind, r.name, r.phase)
running = list_jobs(phase="Running") # or "running"; matched case-insensitively
get_job("tj-abc12").stop()A Run holds an authenticated client, so it can't be pickled or deep-copied. To act on a job from another process, pass the job name and get a new handle there with get_job(name).
Watching progress
wait() prints nothing by default. In a script or CI log, pass progress=True to print a timestamped line per phase change and a last line with the total time:
run.wait(progress=True)
# [12:00:01] run tj-abc12: Pending
# [12:00:06] run tj-abc12: Pending -> Running
# [12:11:43] run tj-abc12: Running -> Succeeded
# [12:11:43] run tj-abc12: Succeeded after 702.0s (terminal)To handle changes yourself, pass a function as on_change=. wait() calls it with the new phase on every change it sees, including the move to a terminal phase. The phase wait() sees when it starts doesn't count as a change. So if the job is already Running, your function is called once, on Succeeded. An exception in your function propagates out of wait():
run.wait(on_change=lambda phase: print("now:", phase))In a notebook, run.watch() shows a status table that updates on each change. It takes the same timeout, poll_interval and on_unknown_phase arguments as wait(). In a plain script, or a kernel without IPython, it falls back to wait(progress=True):
run.watch()A Run or RunList also shows as a table when it is the last expression in a cell. The table makes no request. It shows the status the handle last fetched, or ? in every status column if it has never fetched one:
run # one run, as a field and value table
list_jobs(phase="Running") # one row per runSlicing a RunList, or adding two together, gives another RunList, which keeps the table. sorted() and the other builtins that copy a list return a plain list, which doesn't.
When a run fails
A failed run carries a reason and a message. run.failure_reason is one of these:
| Reason | What happened |
|---|---|
JobFailed |
Your entrypoint exited non-zero, or Ray ended the job. The message holds the end of its output, where the error usually is. |
GPUUnavailable |
A GPU the job was given could not be prepared for 5 minutes, so the platform failed the job. The message is the GPU driver's. Submit again on other GPUs with gpu_placements, and tell your account team which GPU it was. |
ValidationFailed |
Ray refused the job before it started. The message says which setting. |
DeploymentFailed |
The job's Ray cluster could not be started. |
run.failure_message keeps the last 8 KiB of the failure detail. For a long traceback that is the end, with the exception. For the whole output, read the log.
Logs
The platform saves everything your entrypoint prints, stdout and stderr, to a file on the first Volume the job mounts writable:
<mount_path>/.privatemind/training/<job name>/driver.logrun.log_path gives that path as the job sees it. The file is on your Volume, so it outlives the job and its Ray cluster. A job that mounts no writable Volume has no log file, and run.log_path is None.
pm.finetune and @pm.train mount your home Volume first. In a workspace on that Volume, the log of job tj-abc12 is at ~/.privatemind/training/tj-abc12/driver.log.
run.logs() returns the file's contents when the SDK can read it, as in a workspace on the same home Volume: a path under /workspace is looked for under your home Volume too. Anywhere else it returns run.failure_message, and an empty string when there is nothing to show yet.
run = get_job("tj-abc12")
if run.phase == "Failed":
print(run.failure_reason, run.log_path)
print(run.logs())Stop or delete a run
run.stop() ends the job and releases its GPUs. The run's record stays, with the phase Stopped, and so does everything the job wrote to your Volumes. A stopped run can't start again. Submit a new one instead, and if the run saved checkpoints, resume from one. run.cancel() does the same as stop(). Before SDK 0.11.0, cancel() deleted the run.
run.delete() ends the job if it still runs and removes its record, so get_job no longer finds it. Files on your Volumes, the log included, stay.
Stopping or deleting a run that is already gone is not an error. Stopping one that has already ended raises ConflictError.
You can stop and delete your own runs. Org admins can stop and delete any run in the org.
Reading results back
run.metrics(), run.params() and run.list_artifacts() read the job's MLflow run, so you can pull results into the notebook without opening the MLflow UI. They only read.
run.wait()
print(run.metrics()) # {'loss': 0.21, 'accuracy': 0.94}
print(run.params()) # {'lr': '0.001', 'epochs': '10'}
for a in run.list_artifacts():
print(a.path, a.size if not a.is_dir else "(dir)")list_artifacts() returns RunArtifact entries with path, is_dir and size (None for a directory). RunArtifact is the SDK's own type, so it doesn't change when MLflow does.
These need the mlflow extra (pip install 'privatemind[mlflow]') and a tracking URI. The SDK takes the URI from the tracking_uri= argument, then from PRIVATEMIND_MLFLOW_URI, and raises ConfigError if neither is set. Until a job starts tracking, run.mlflow_run_id is None and all three raise PrivatemindError, so wait for the run or check the id first.
For anything else, mlflow_client() returns a real MlflowClient pointed at the same URI:
from privatemind import mlflow_client
client = mlflow_client() # or mlflow_client(tracking_uri=...)
history = client.get_metric_history(run.mlflow_run_id, "loss")Completion webhooks
The platform can POST to a URL of yours when a job succeeds or fails, so you don't have to poll with run.wait(). Pass webhook_url when you submit, and webhook_secret to sign the callback:
run = submit_training(
entrypoint="python train.py",
image="<training-image>",
workers=1,
target_cluster="<your-gpu-cluster>",
webhook_url="https://hooks.acme.com/pmind/training",
webhook_secret="whsec_...", # optional; enables signing
)webhook_url must be https. The SDK rejects anything else at submit, localhost included. Your org admin must also register the host as an approved egress (outbound) destination, or the submit is rejected (see the prerequisite below).
The platform sends one POST per job. Delivery is at least once, so a retry can send the same POST twice: drop duplicates by delivery_id.
{
"job_id": "tj-abc12",
"name": "tj-abc12",
"delivery_id": "9f1c...",
"status": "Succeeded",
"started_at": "2026-01-02T10:00:00Z",
"finished_at": "2026-01-02T10:14:30Z"
}status is the terminal phase, Succeeded or Failed. Treat two fields as optional: a Failed body also carries an error string, and mlflow_run_id appears only once the run has one.
Verifying the signature
When you set webhook_secret, every delivery carries Standard Webhooks headers:
webhook-id: thedelivery_id(the same across retries of one delivery).webhook-timestamp: unix seconds.webhook-signature:v1,<base64 HMAC-SHA256(secret, "{webhook-id}.{webhook-timestamp}.{raw-body}")>.
Verify with any Standard Webhooks library, or directly:
import base64, hashlib, hmac
def verify(secret, headers, raw_body):
signed = f"{headers['webhook-id']}.{headers['webhook-timestamp']}.{raw_body}"
expected = base64.b64encode(
hmac.new(secret.encode(), signed.encode(), hashlib.sha256).digest()
).decode()
got = headers["webhook-signature"].removeprefix("v1,")
return hmac.compare_digest(expected, got)Verify over the raw request body. Parsing the JSON and dumping it again changes the bytes and breaks the signature. To guard against replays, reject deliveries whose webhook-timestamp is far from the current time.
Never pass a webhook_secret or webhook_url with leading or trailing whitespace. The server strips the secret before signing. If your receiver keeps a stray newline, it checks against a different value and every signature fails, with no error. An all-whitespace secret strips to empty, and the submit is rejected with 400. A secret read from a file usually ends in a newline: strip it with Path("webhook.secret").read_text().strip(). The SDK rejects padded values at submit with ValidationError.
Checking delivery from the SDK
run.webhook_status shows where a delivery stands, as the server reports it:
| Value | Meaning |
|---|---|
not_configured |
no webhook_url was set |
disabled |
a webhook is configured but delivery is off in this environment |
pending |
queued or mid-retry |
blocked_egress |
the host is not an approved egress destination for your org; nothing is delivered until an admin registers it |
delivered |
the callback was accepted (2xx) |
failed |
retries exhausted |
unknown |
the server could not read the delivery state; check again shortly |
The value comes from the last fetched status, so call refresh() first, or use get_job, which fetches:
from privatemind import get_job
print(get_job("tj-abc12").webhook_status) # e.g. "delivered"It is None only if the server leaves the field out. Treat any value you don't recognise as unknown.
Prerequisite: the host must be allowlisted
Webhook deliveries leave PrivateMind only through your org's egress proxy, which reaches only the hosts your org has registered. So your org admin registers each webhook host once as an egress destination, and every later job can use it. A submit that names an unregistered host is rejected outright with 400, and nothing is queued. blocked_egress is the other case: the host was registered when you submitted but removed before the job finished. The delivery then waits until the host is registered again.
Promote a notebook function
@pm.train takes a function you've tested in the notebook and runs it as a training job tracked in MLflow, without leaving Python.
import privatemind as pm
@pm.train(workers=1, gpus_per_worker=1)
def train(lr=3e-4, epochs=10):
import mlflow, torch # re-resolved on the worker, must be in the image
assert torch.cuda.is_available()
...
train(lr=1e-3) # runs locally in the notebook (validate)
run = train.promote(lr=1e-3) # packages + submits to the GPU -> Run
run.wait(); print(run.mlflow_run_id)- Calling the function, or
train.local(...), runs it in the notebook. train.promote(...)packages the function with its arguments, copies it to your home Volume and submits it throughsubmit_training. It returns aRun.train.with_options(gpus_per_worker=4)returns a copy with changed settings, so you don't have to decorate the function again.
The decorator takes the same arguments as submit_training, except entrypoint, namespace and the two webhook fields. It adds two of its own: experiment= groups the MLflow runs under a named experiment, and home_volume= sets the Volume that carries your code.
In a workspace, target_cluster and home_volume default from the workspace, and image comes from the platform's allowlist, so you usually pass only workers=1 and gpus_per_worker. Outside a workspace the image is found the same way, but target_cluster and home_volume have no default: pass both, or promote() raises ConfigError.
Finding the image needs SDK 0.4.0 or newer. Older versions look for a PRIVATEMIND_TRAINING_IMAGE variable the platform doesn't set, and raise ConfigError naming it. Pass image= yourself, or upgrade with pip install -U privatemind.
Three rules make a function promotable:
- Take all inputs as arguments. The function and its arguments travel to the job as one serialised payload. A notebook global in the function, such as a DataFrame or a loaded model, either fails with
ValidationError(the function "isn't self-contained") or bloats the payload. The SDK warns when the payload passes about 5 MB. Read data from mounted Volumes, and write outputs to a Volume or to MLflow. - Import inside the function. Imports run again on the worker against the training image, not the notebook's installed packages, so import only what the image provides.
- GPU code runs on a GPU worker. With
gpus_per_workerabove 0,promote()runs your function on a GPU worker, so plaintorchcode that uses CUDA works as written.
If your org has MLflow set up, every promote is tracked unless you pass mlflow=False:
- The run is named after the job, which is named after your function unless you pass
name=. - CPU, GPU and memory system metrics are recorded.
- Anything you log inside the function, such as metrics, artifacts or
mlflow.pytorch.log_model(...), goes to that run.
If the job can't reach MLflow when it starts, the function runs untracked and the job's log says so.
To pull the results into the notebook afterwards, see Reading results back.
Mounting data
volumes mounts existing Volumes into every worker. Each entry is a mapping with these keys:
volume: the Volume's name.mount_path: an absolute path, unique within the job.read_only: optional, defaultFalse. It must be a real boolean, not a string.
run = submit_training(
entrypoint="python train.py --data /data --out /artifacts",
image="<training-image>",
workers=1,
target_cluster="<your-gpu-cluster>",
gpus_per_worker=1,
volumes=[
{"volume": "datasets", "mount_path": "/data", "read_only": True},
{"volume": "checkpoints", "mount_path": "/artifacts"},
],
)The SDK rejects relative or duplicate mount paths before submitting. With @pm.train, your home Volume is always mounted as well, because it carries the promoted code.
Mounting registry weights
weights mounts versions from your org's registry read-only into the job, up to 8, each given as <name>:<version>. A version goes in /home/privatemind/models/<folder>, the folder named after the last part of the model's name. To choose the folder, pass a mapping with the ref and a mount_path, as here:
run = submit_training(
entrypoint="python train.py --base /home/privatemind/models/base",
image="<training-image>",
workers=1,
target_cluster="<your-gpu-cluster>",
gpus_per_worker=8,
weights=[{"ref": "credit-ft:2026-11-16", "mount_path": "/home/privatemind/models/base"}],
)The folder takes lowercase letters, digits and dashes, up to 63 characters. This mapping differs from the one workspace jobs take, which names the folder with name: a training job ignores name. The platform pins each version to its digest when you submit.
Explicit client
For a longer session, or to manage configuration yourself, use a Client as a context manager. Both URLs default to the production platform, so pass them only to point somewhere else:
from privatemind import Client
with Client(
base_url="https://api.privatemind.com", # platform API (images, batch)
app_url="https://privatemind.com", # app backend (training jobs)
token="PMIND...:abcdef...",
) as pm:
run = pm.submit_training(
entrypoint="python train.py",
image="<training-image>",
workers=1,
target_cluster="<your-gpu-cluster>",
)
for r in pm.list_jobs():
print(r.name, r.phase)The module-level functions (submit_training, list_jobs, get_job) share one client per process, created from the environment on first use. reconfigure(base_url=..., app_url=..., token=...) replaces it, which helps in a notebook when credentials change. Each function also takes client= to send a single call through a client of your own.
Each error has its own type (AuthError, ConfigError, ValidationError, ForbiddenError, NotFoundError, ConflictError, RateLimitError, ServerError, UnknownPhaseError), all subclasses of PrivatemindError. The SDK overview covers them in full.
From a shell
The privatemind command that comes with the SDK lists, reads and stops training runs. It authenticates the same way as the SDK: in a workspace it is ready to use, and elsewhere it reads PRIVATEMIND_TOKEN.
| Command | What it does |
|---|---|
privatemind jobs ls |
Your training runs, with phase, start and end times, and failure reason. |
privatemind jobs get <name> |
One run: its phase, times, MLflow run, failure reason, log path, link and failure message. |
privatemind jobs logs <name> |
The run's log, as run.logs() returns it. |
privatemind jobs stop <name> |
Stop the run and keep its record. |
Where next
- Compute & training: what the platform offers and how it fits together.
- Fine-tuning:
pm.finetunedoes all of this in one call, for SFT, DPO, KTO and GRPO. - Evaluate and publish: score a fine-tuned model, push it to your registry and serve it.
- SDK overview: configuration, authentication, and the full error catalog.
- Workspaces: the GPU notebook environment where the SDK is set up for you.
- API keys: create the key the SDK authenticates with outside a workspace.