Training jobs

Training jobs are part of the PrivateMind Python SDK, which ships pre-installed and pre-configured in every GPU workspace. In a workspace there is nothing to install and nothing to set up: import privatemind and call submit_training. Your token, the API URLs, and your GPU cluster name are injected as environment variables, so there is no key to paste and no endpoint to configure. Notebooks, scripts, and CI all use the same interface.

Authentication

In a workspace, skip this section: credentials and URLs are injected, so nothing here needs setting up.

Outside a workspace (a laptop, CI), the SDK resolves credentials in order:

1. token= passed to Client(...)
2. the PRIVATEMIND_TOKEN environment variable
3. ~/.privatemind/auth (must be mode 0600, owned by you, and not a symlink)

Use a PrivateMind API key as the token. The SDK talks to two services, and training uses the second:

• Platform API: base_url= or PRIVATEMIND_URL, defaulting to https://api.privatemind.com. Serves image generation and batch.
• App backend: app_url= or PRIVATEMIND_APP_URL, defaulting to https://privatemind.com. Serves the training methods (submit_training, list_jobs, get_job).

Both must be https://. The full configuration story lives on the SDK overview.

Submit a job

submit_training is the core entry point. It is keyword-only and returns a Run handle.

from privatemind import submit_training

run = submit_training(
    entrypoint="python train.py --epochs 10",
    image="<training-image>",
    workers=1,
    target_cluster="<your-gpu-cluster>",
    gpus_per_worker=1,
)

print(run)            # Run(name='tj-abc12', phase='Pending')
print(run.url)        # link to the run in the PrivateMind UI
run.wait()            # block until the job is terminal
print(run.phase, run.mlflow_run_id)

The platform runs only its own training images, so <training-image> is not an arbitrary reference. On SDK 0.4.0 or newer, Client().list_training_images() returns the allowlist, default first; pass an entry's value through unchanged, digest and all. The @pm.train decorator does that lookup for you.

Arguments are validated client-side, so a bad value (workers=0, a malformed name, a reserved env key) raises ValidationError before any request is made.

GPUs

Set gpus_per_worker and leave gpu_placements unset: the platform derives a placement from the GPUs your org owns that are free right now and pins the job to them. This is the usual path; you do not need to know which physical GPUs your org has.

To pin exact devices instead, pass explicit placements: one placement per worker, each a mapping with a host and exactly gpus_per_worker GPU indices (integers 0 to 7, no duplicates):

gpu_placements=[{"host": "<gpu-host>", "indices": [0, 1]}]   # workers=1, gpus_per_worker=2

The SDK checks the shape at submit (gpu_placements requires gpus_per_worker above 0, and the list length must equal workers); the platform is the authority on ownership, index conflicts, and quota, and rejects a job that asks for GPUs it cannot have. The platform accepts a single explicit placement per job, so pin only single-worker jobs; for multi-worker GPU jobs, omit gpu_placements and let the platform place the workers.

One thing to know about where your code runs: the entrypoint executes on the job's Ray head, which holds no GPU. The GPUs live on the workers, so a training script reaches them through Ray (Ray Train, or ray.remote(num_gpus=...)) as in any Ray program. The @pm.train decorator handles this dispatch for you.

The Run handle

submit_training returns a Run, a live handle to the job. Its identity (name, namespace, url) is fixed; its status is a snapshot that you refresh. Status properties serve the last fetched snapshot and fetch one automatically on first access.

A job reports one of six phases. The SDK groups them into two sets, both importable from privatemind:

wait() polls while the phase is in ACTIVE_PHASES and returns as soon as it is not. The poll interval starts at poll_interval seconds and doubles with each poll, capped at 30 seconds (or at your initial interval, if you set one higher). timeout=None waits forever; with a timeout, a job still running when it expires raises TimeoutError. Timing out only stops the watching: the job keeps running until it finishes or you cancel it.

A phase in neither set raises UnknownPhaseError, which carries .run_name and .phase. It means the platform has grown a phase your installed SDK cannot place, so the SDK refuses to report the job as finished when it cannot tell: upgrade privatemind, or read run.phase and decide for yourself. wait(on_unknown_phase="return") returns instead and emits a UserWarning, but prefer the default. A notebook that opens with warnings.filterwarnings("ignore") turns that into a silent early return, and run.metrics() will then serve you half a training run's numbers with nothing on screen to say so.

try:
    run.wait(timeout=3600)
except TimeoutError:
    run.cancel()

if run.phase == "Succeeded":
    print(run.mlflow_run_id, run.end_time)

List or look up jobs without holding the original handle. list_jobs() returns a RunList of handles with their status already populated: a Run for each training run and a Job for each workspace job, with kind ("training" or "job") on every item to tell them apart. phase= narrows it on an exact, case-insensitive match applied client-side to both kinds. list_training_jobs() returns the training runs alone. get_job(name) stays the training lookup; a workspace job is get_workspace_job(name):

from privatemind import list_jobs, get_job

for r in list_jobs():                   # training runs and workspace jobs you may see
    print(r.kind, r.name, r.phase)

running = list_jobs(phase="Running")    # or "running"; matched case-insensitively

get_job("tj-abc12").cancel()

One caveat if you are pinned to an older SDK: list_jobs() came back empty against the platform on every release before 0.5.0, because it read the wrong key out of the response and parsed none of the rows. Upgrade before you rely on it.

Watching progress

wait() is silent by default. progress=True prints one timestamped line per phase transition and a final line with the elapsed total, which is what you want in a script or a CI log:

run.wait(progress=True)
# [12:00:01] run tj-abc12: Pending
# [12:00:06] run tj-abc12: Pending -> Running
# [12:11:43] run tj-abc12: Running -> Succeeded
# [12:11:43] run tj-abc12: Succeeded after 702.0s (terminal)

on_change= hands the transitions to a callable of yours instead, called with the new phase string on every transition the poll observes, including the one into the terminal phase. The status wait() fetches on entry is the baseline and does not fire it, so waiting on a job that is already Running calls back once, on Succeeded. An exception raised inside the callback propagates out of wait():

run.wait(on_change=lambda phase: print("now:", phase))

In a notebook, run.watch() renders a status table that updates in place on each transition and settles on the final status. It takes the same timeout, poll_interval and on_unknown_phase arguments as wait(), and where there is no live IPython display (a plain script, or a kernel without IPython installed) it falls back to wait(progress=True):

run.watch()

A Run, and the RunList that list_jobs() returns, also render as tables on their own when either is the last expression in a cell. That rendering reads the status already on the handle and never makes a request, so a handle you have not refreshed shows what it last saw, and one that has never fetched shows ? in every status column:

run                           # one run, as a field and value table
list_jobs(phase="Running")    # one row per run

Slicing a RunList, or adding two of them together, gives you another RunList and keeps that rendering. sorted() and the other builtins that copy a list return a plain list, which does not.

Reading results back

run.metrics(), run.params() and run.list_artifacts() read the MLflow run behind a job, so you can pull results into the notebook without opening the MLflow UI. They read only; nothing writes back.

run.wait()

print(run.metrics())        # {'loss': 0.21, 'accuracy': 0.94}
print(run.params())         # {'lr': '0.001', 'epochs': '10'}

for a in run.list_artifacts():
    print(a.path, a.size if not a.is_dir else "(dir)")

list_artifacts() returns RunArtifact entries with path, is_dir, and size (None for a directory). It is a plain dataclass rather than an MLflow type, so the SDK surface does not shift under you when MLflow changes.

These need the mlflow extra (pip install 'privatemind[mlflow]') and a tracking URI. The URI resolves from the tracking_uri= argument first, then PRIVATEMIND_MLFLOW_URI; with neither you get a ConfigError telling you so. A job that has not started tracking yet has no run.mlflow_run_id, and all three raise PrivatemindError in that case, so wait for the run or check the id first.

For anything past these three, mlflow_client() hands you a real MlflowClient pointed at the same resolved URI:

from privatemind import mlflow_client

client = mlflow_client()                  # or mlflow_client(tracking_uri=...)
history = client.get_metric_history(run.mlflow_run_id, "loss")

Completion webhooks

Instead of polling with run.wait(), you can have the platform POST to a URL of yours when a job succeeds or fails. Pass webhook_url (and, to sign the callback, webhook_secret) on submit:

run = submit_training(
    entrypoint="python train.py",
    image="<training-image>",
    workers=1,
    target_cluster="<your-gpu-cluster>",
    webhook_url="https://hooks.acme.com/pmind/training",
    webhook_secret="whsec_...",   # optional; enables signing
)

webhook_url must be https; the SDK rejects anything else at submit, with no localhost exception. Before it works, your org admin has to register the host as an approved egress destination (see the prerequisite below); until then, submit is rejected.

When the job succeeds or fails, the platform sends one POST. Delivery is at-least-once: a retry can repeat a delivery, so treat delivery_id as an idempotency key and dedupe on it.

{
  "job_id": "tj-abc12",
  "name": "tj-abc12",
  "delivery_id": "9f1c...",
  "status": "Succeeded",
  "started_at": "2026-01-02T10:00:00Z",
  "finished_at": "2026-01-02T10:14:30Z"
}

status is the terminal phase, Succeeded or Failed. Two fields are conditional, so treat them as optional: a Failed body also carries an error string, and mlflow_run_id appears only once the run has one assigned. For the platform-side view of training, see Compute & training.

Verifying the signature

When you set webhook_secret, every delivery carries Standard Webhooks headers:

• webhook-id: the delivery_id (stable across retries of the same delivery).
• webhook-timestamp: unix seconds.
• webhook-signature: v1,<base64 HMAC-SHA256(secret, "{webhook-id}.{webhook-timestamp}.{raw-body}")>.

Verify with any Standard Webhooks library, or directly:

import base64, hashlib, hmac

def verify(secret, headers, raw_body):
    signed = f"{headers['webhook-id']}.{headers['webhook-timestamp']}.{raw_body}"
    expected = base64.b64encode(
        hmac.new(secret.encode(), signed.encode(), hashlib.sha256).digest()
    ).decode()
    got = headers["webhook-signature"].removeprefix("v1,")
    return hmac.compare_digest(expected, got)

Sign and verify over the raw request body, not a re-serialised copy: reparsing and re-dumping the JSON changes the bytes and breaks the signature. Reject deliveries whose webhook-timestamp is far from now to blunt replays.

Checking delivery from the SDK

run.webhook_status reports where a delivery stands, server-authoritative:

The value reflects the last fetched status, so refresh() first (or use get_job, which fetches):

from privatemind import get_job

print(get_job("tj-abc12").webhook_status)   # e.g. "delivered"

It is None only when the server response omits the field entirely. Treat any value you do not recognise as unknown rather than failing.

Prerequisite: the host must be allowlisted

The data tier reaches the internet only through a per-org egress proxy confined to hosts your org has registered, so a webhook target is not a free-form URL. Your org admin registers the webhook host once as an egress destination. A submit that names a host which is not a registered destination is rejected outright (400); nothing is queued. The blocked_egress status is the other case: a host that was registered when you submitted but was removed before the job finished, so the pending delivery parks until it is registered again. Either way it is a one-time onboarding step per host, not per job.

Promote a notebook function

@pm.train takes a function you validated interactively and runs it as a distributed, MLflow-tracked job, without leaving Python.

import privatemind as pm

@pm.train(workers=1, gpus_per_worker=1)
def train(lr=3e-4, epochs=10):
    import mlflow, torch       # re-resolved on the worker, must be in the image
    assert torch.cuda.is_available()
    ...

train(lr=1e-3)                 # runs locally in the notebook (validate)
run = train.promote(lr=1e-3)   # packages + submits to the GPU -> Run
run.wait(); print(run.mlflow_run_id)

Calling the function (or train.local(...)) runs it in the notebook. train.promote(...) packages the function with its bound arguments, stages it on your home Volume, and submits it through submit_training, returning the same Run handle as above. train.with_options(workers=4) returns a copy with tweaked settings without redecorating.

The decorator mirrors submit_training's tuning knobs (cpu_per_worker, memory_per_worker, env, mlflow, ray_version, ttl_seconds_after_finished, name, volumes, gpu_placements, client; the webhook fields are submit-only) and adds two of its own: experiment= groups the MLflow runs under a named experiment, and home_volume= overrides the Volume that carries your code. target_cluster and home_volume default from the workspace context, and image is resolved against the platform's allowlist, so in a notebook you usually pass only workers and gpus_per_worker. Outside a workspace the image resolves the same way, but target_cluster and home_volume have no default: pass both, or promote() raises ConfigError.

Resolving the image needs SDK 0.4.0 or newer. Older ones have no lookup to fall back on: they expect a PRIVATEMIND_TRAINING_IMAGE variable that the platform no longer injects, so they raise ConfigError naming it. Either pass image= yourself, or upgrade with pip install -U privatemind.

Three rules make a function promotable:

1. Take all inputs as arguments. The function and its arguments travel as a serialised payload. Closing over a notebook global (a DataFrame, a loaded model) either fails to serialise (ValidationError: the function "isn't self-contained") or bloats the payload (the SDK warns above roughly 5 MB). Read data from mounted Volumes; write outputs to a Volume or to MLflow.
2. Import inside the function. Imports re-resolve on the worker, so import only what the training image provides.
3. GPU code runs on a GPU worker. A GPU promote dispatches your function onto the GPU worker automatically, so plain torch code that uses CUDA works as written.

With workers above 1, the function launches under Ray Train with the torch.distributed world set up (RANK, WORLD_SIZE, MASTER_ADDR), so DDP and DeepSpeed initialisation inside the function work across workers. Rank 0 owns the MLflow run; guard anything you want logged once on the world rank.

Every promote is tracked in MLflow with no setup: the run is named after the job (which itself is named after your function unless you pass name=), CPU/GPU/memory system metrics are captured automatically, and anything you log inside the function (metrics, artifacts, mlflow.pytorch.log_model(...)) lands on that run. Pass experiment="my-experiment" to group runs, or mlflow=False to turn tracking off. To pull the results back into the notebook afterwards, see Reading results back.

Mounting data

volumes mounts existing Volumes into every worker. Each entry is a mapping with a volume name, an absolute mount_path, and an optional read_only flag (default False, and it must be a real boolean, not a string):

run = submit_training(
    entrypoint="python train.py --data /data --out /artifacts",
    image="<training-image>",
    workers=1,
    target_cluster="<your-gpu-cluster>",
    gpus_per_worker=1,
    volumes=[
        {"volume": "datasets", "mount_path": "/data", "read_only": True},
        {"volume": "checkpoints", "mount_path": "/artifacts"},
    ],
)

Mount paths must be absolute and unique within the job; the SDK rejects relative or duplicate paths before submit. Every worker sees the same mounts. On @pm.train, entries you pass are mounted in addition to your home Volume, which always rides along to carry the promoted code.

Explicit client

For a longer session, or to manage configuration yourself, use an explicit Client as a context manager. Both URLs default to the production platform; pass them only to target something else:

from privatemind import Client

with Client(
    base_url="https://api.privatemind.com",   # platform API (always required)
    app_url="https://privatemind.com",        # app backend (training jobs)
    token="PMIND...:abcdef...",
) as pm:
    run = pm.submit_training(
        entrypoint="python train.py",
        image="<training-image>",
        workers=4,
        target_cluster="<your-gpu-cluster>",
    )
    for r in pm.list_jobs():
        print(r.name, r.phase)

The module-level functions (submit_training, list_jobs, get_job) share one lazily created, process-wide client configured from the environment. reconfigure(base_url=..., app_url=..., token=...) swaps it, which is useful in a notebook when credentials change; each function also accepts client= to route a single call through an explicit client.

Errors are typed (AuthError, ConfigError, ValidationError, ForbiddenError, NotFoundError, ConflictError, RateLimitError, ServerError, UnknownPhaseError), all under a common PrivatemindError, so you can catch precisely. See the SDK overview for the full error section.

Where next

• Compute & training: what the platform offers and how it fits together.
• Fine-tuning: pm.finetune wraps all of this into one call for LoRA and QLoRA SFT.
• SDK overview: configuration, authentication, and the full error catalog.
• Workspaces: the GPU notebook environment where the SDK is pre-configured.
• API keys: create the key the SDK authenticates with outside a workspace.