Get API key

Training jobs

View as Markdown

Submit and manage training jobs from any notebook, script or shell.

Note

The SDK uses your org's GPU allocation. If your org doesn't have one, training calls return an error. See Compute & training.

The PrivateMind Python SDK submits and manages training jobs, with the same calls from a notebook, a script or CI. It comes installed in every workspace, with your token, the API URLs and your GPU cluster name already set as environment variables. There, you import privatemind and call submit_training.

Failure reasons, logs, stopping a run, registry weights and the privatemind jobs command need SDK 0.11.0 or later: pip install -U "privatemind>=0.11".

Authentication

In a workspace you can skip this section, because credentials and URLs are already set.

Outside a workspace, on a laptop or in CI, the SDK looks for a token in this order:

  1. token= passed to Client(...)
  2. the PRIVATEMIND_TOKEN environment variable
  3. ~/.privatemind/auth (must be mode 0600, owned by you, and not a symlink)

The token is a PrivateMind API key. The SDK talks to two services, and training uses the second:

  • Platform API: base_url= or PRIVATEMIND_URL, default https://api.privatemind.com. Serves image generation and batch.
  • App backend: app_url= or PRIVATEMIND_APP_URL, default https://privatemind.com. Serves the training methods (submit_training, list_jobs, get_job).

Both must use https://. The SDK overview covers the rest of the configuration.

Submit a job

submit_training starts a job and returns a Run handle. All its arguments are keyword-only.

Python
from privatemind import submit_training

run = submit_training(
    entrypoint="python train.py --epochs 10",
    image="<training-image>",
    workers=1,
    target_cluster="<your-gpu-cluster>",
    gpus_per_worker=1,
)

print(run)            # Run(name='tj-abc12', phase='Pending')
print(run.url)        # link to the run in the PrivateMind UI
run.wait()            # block until the job is terminal
print(run.phase, run.mlflow_run_id)

image must be one of the platform's own training images. The platform refuses any other image, your own included. There are two: the default PyTorch image, and an image that adds vLLM, which GRPO and evaluation run on. On SDK 0.4.0 or newer, Client().list_training_images() lists every offered version of both, with the default first. Pass an entry's value unchanged, digest included.

Parameter Default Purpose
entrypoint required Shell command the job runs on its Ray head, e.g. python train.py --epochs 10.
image required Container image the workers run, from the platform's list (see above).
workers required Number of Ray workers. Training is supported on one node: use 1, and set the GPUs with gpus_per_worker. The SDK refuses more than 64, or 16 for a job with GPUs.
target_cluster required Your org's GPU cluster name (from your dashboard or account team).
gpus_per_worker 0 GPUs per worker, 0 to 8. 0 is CPU-only.
cpu_per_worker none CPU request per worker: an int or a Kubernetes quantity string ("2", "500m").
memory_per_worker none Memory request per worker, same format ("8Gi").
env none Environment variables for the workers. Keys starting with MLFLOW_ are reserved by the platform and rejected.
mlflow True Track the run in MLflow, which fills in run.mlflow_run_id. If your org has no MLflow set up, pass False or the job is refused.
ray_version platform default Pin the Ray version the job runs.
ttl_seconds_after_finished 600 Seconds the job's Ray cluster stays up after the job ends, whether it succeeded or failed, 0 to 86400 (one day). The platform then removes the cluster. The run's record stays until you delete it.
name assigned Job name: lowercase letters, digits and hyphens, up to 40 characters, starting and ending with a letter or digit. Omit it and the platform assigns one.
namespace assigned Advanced override. Omit it in normal use.
volumes none Volumes mounted into every worker. See Mounting data.
weights none Versions from your org's registry to mount read-only. See Mounting registry weights.
gpu_placements auto-derived Pin exact GPUs. Omit it and the platform picks them. See GPUs.
webhook_url none HTTPS URL the platform POSTs to when the job finishes. See Completion webhooks.
webhook_secret none Signs each webhook delivery. Requires webhook_url.
client shared client An explicit Client to submit through.

The SDK checks arguments before it sends anything. A bad value, such as workers=0, a malformed name or a reserved env key, raises ValidationError.

A run's RAM and vCPU count against your personal compute allowance, which workspaces, inference deployments and jobs also use. The total covers every worker plus the job's Ray head, the node that runs your entrypoint. See Personal compute quotas.

Tip

In a workspace, os.environ["PRIVATEMIND_TARGET_CLUSTER"] holds your cluster name. The @pm.train decorator fills in target_cluster and image for you.

GPUs

Usually you set gpus_per_worker and leave gpu_placements unset. The platform then pins the job to free GPUs your org owns, so you don't need to know which ones you have.

To pin exact devices, pass one placement per worker. Each is a mapping with a host and exactly gpus_per_worker GPU indices (integers 0 to 7, no duplicates):

Python
gpu_placements=[{"host": "<gpu-host>", "indices": [0, 1]}]   # workers=1, gpus_per_worker=2

The SDK checks the shape when you submit: gpu_placements needs gpus_per_worker above 0, and the list length must equal workers. The platform checks ownership, index conflicts and quota, and rejects a job that asks for GPUs it cannot have.

Your entrypoint runs on the job's Ray head, which has no GPU. The GPUs are on the workers, so your script reaches them through Ray (Ray Train, or ray.remote(num_gpus=...)) as in any Ray program. The @pm.train decorator does this for you.

The Run handle

A Run is a live handle to the job. Its name, namespace and url never change. Its status is a snapshot. Status properties such as run.phase return the last status fetched, and fetch it the first time you read one. Call run.refresh() to fetch it again.

Member What it gives you
run.phase Lifecycle state: Pending, Queued or Running while the job is live, then Succeeded, Failed, Stopped or Terminating.
run.url Link to the run in the PrivateMind UI.
run.failure_reason Why a failed run failed, as one word, else None. See When a run fails.
run.failure_message The last 8 KiB of the failure detail, else None.
run.log_path Where the job writes its full log, as a path inside the job. None when the job mounts no writable Volume.
run.logs() The job's driver log, or the failure message where the log can't be read. See Logs.
run.mlflow_run_id MLflow run id once tracking has started, else None.
run.ray_job_name Underlying Ray submission id once the job is admitted, else None.
run.start_time, run.end_time RFC 3339 timestamps, None until the job starts or finishes.
run.webhook_status Completion-webhook delivery state. See Checking delivery from the SDK.
run.refresh() Fetch the status again. Returns the run.
run.wait(timeout=None, poll_interval=5.0) Block until the job stops running. Returns the run.
run.watch(timeout=None, poll_interval=5.0) Block like wait(), showing a live status table in a notebook. Returns the run.
run.stop() End the job and keep its record. See Stop or delete a run.
run.cancel() The same as stop().
run.delete() End the job if it still runs, and remove its record.
run.publish(name, version=None) Push a fine-tune's result to your org's registry and return <name>:<version>. See Evaluate and publish.
run.metrics() Latest metric values from this run's MLflow run, as {name: float}.
run.params() Params logged to this run's MLflow run, as {name: str}.
run.list_artifacts(path=None) Artifact entries under the run, optionally rooted at path.

A job reports one of seven phases. The SDK groups them into two sets, both importable from privatemind:

Phase Set Meaning
Pending ACTIVE_PHASES the job was accepted; the platform has not started on it yet
Queued ACTIVE_PHASES the workers exist and are waiting for GPUs to come free
Running ACTIVE_PHASES the workers are up and your entrypoint is executing
Succeeded TERMINAL_PHASES the job exited cleanly
Failed TERMINAL_PHASES the job exited non-zero, or the platform could not run it. run.failure_reason says which
Stopped TERMINAL_PHASES you or an admin stopped the job
Terminating TERMINAL_PHASES tear-down is under way. You are unlikely to see it: the platform doesn't report it, and a deleted job stops answering. The SDK counts it as terminal so a job that reaches it ends the wait

wait() polls while the phase is in ACTIVE_PHASES and returns as soon as it isn't.

  • poll_interval is the first gap between polls, in seconds. It doubles each poll, up to 30 seconds, or up to poll_interval if you set it higher.
  • timeout=None waits forever. If the job is still running when a timeout expires, wait() raises TimeoutError. The job keeps running until it finishes or you cancel it.

A phase in neither set raises UnknownPhaseError, which carries .run_name and .phase. It means the platform reports a phase your installed SDK doesn't know. The SDK then can't tell whether the job has finished, so it won't say it has. Upgrade privatemind, or read run.phase and decide yourself. SDK versions before 0.11.0 don't know Stopped, so waiting on a stopped run raises this error there.

wait(on_unknown_phase="return") returns with a UserWarning and doesn't raise, but keep the default. If your notebook runs warnings.filterwarnings("ignore"), that return is silent. run.metrics() then gives you half a training run's numbers with nothing on screen to say so.

Python
try:
    run.wait(timeout=3600)
except TimeoutError:
    run.stop()

if run.phase == "Succeeded":
    print(run.mlflow_run_id, run.end_time)
elif run.phase == "Failed":
    print(run.failure_reason)
    print(run.logs())

You can list or look up jobs without the original handle:

  • list_jobs() returns a RunList of handles with their status filled in: a Run for each training run and a Job for each workspace job. Each item's kind is "training" or "job". On SDK versions before 0.5.0 it always returns an empty list, so upgrade first.
  • phase= keeps only jobs in that phase, for both kinds. The SDK does the filtering, on an exact match that ignores case.
  • list_training_jobs() returns only the training runs.
  • get_job(name) looks up a training run, and get_workspace_job(name) a workspace job.
Python
from privatemind import list_jobs, get_job

for r in list_jobs():                   # training runs and workspace jobs you may see
    print(r.kind, r.name, r.phase)

running = list_jobs(phase="Running")    # or "running"; matched case-insensitively

get_job("tj-abc12").stop()
Note

A Run holds an authenticated client, so it can't be pickled or deep-copied. To act on a job from another process, pass the job name and get a new handle there with get_job(name).

Watching progress

wait() prints nothing by default. In a script or CI log, pass progress=True to print a timestamped line per phase change and a last line with the total time:

Python
run.wait(progress=True)
# [12:00:01] run tj-abc12: Pending
# [12:00:06] run tj-abc12: Pending -> Running
# [12:11:43] run tj-abc12: Running -> Succeeded
# [12:11:43] run tj-abc12: Succeeded after 702.0s (terminal)

To handle changes yourself, pass a function as on_change=. wait() calls it with the new phase on every change it sees, including the move to a terminal phase. The phase wait() sees when it starts doesn't count as a change. So if the job is already Running, your function is called once, on Succeeded. An exception in your function propagates out of wait():

Python
run.wait(on_change=lambda phase: print("now:", phase))

In a notebook, run.watch() shows a status table that updates on each change. It takes the same timeout, poll_interval and on_unknown_phase arguments as wait(). In a plain script, or a kernel without IPython, it falls back to wait(progress=True):

Python
run.watch()

A Run or RunList also shows as a table when it is the last expression in a cell. The table makes no request. It shows the status the handle last fetched, or ? in every status column if it has never fetched one:

Python
run                           # one run, as a field and value table
list_jobs(phase="Running")    # one row per run

Slicing a RunList, or adding two together, gives another RunList, which keeps the table. sorted() and the other builtins that copy a list return a plain list, which doesn't.

When a run fails

A failed run carries a reason and a message. run.failure_reason is one of these:

Reason What happened
JobFailed Your entrypoint exited non-zero, or Ray ended the job. The message holds the end of its output, where the error usually is.
GPUUnavailable A GPU the job was given could not be prepared for 5 minutes, so the platform failed the job. The message is the GPU driver's. Submit again on other GPUs with gpu_placements, and tell your account team which GPU it was.
ValidationFailed Ray refused the job before it started. The message says which setting.
DeploymentFailed The job's Ray cluster could not be started.

run.failure_message keeps the last 8 KiB of the failure detail. For a long traceback that is the end, with the exception. For the whole output, read the log.

Logs

The platform saves everything your entrypoint prints, stdout and stderr, to a file on the first Volume the job mounts writable:

Text
<mount_path>/.privatemind/training/<job name>/driver.log

run.log_path gives that path as the job sees it. The file is on your Volume, so it outlives the job and its Ray cluster. A job that mounts no writable Volume has no log file, and run.log_path is None.

pm.finetune and @pm.train mount your home Volume first. In a workspace on that Volume, the log of job tj-abc12 is at ~/.privatemind/training/tj-abc12/driver.log.

run.logs() returns the file's contents when the SDK can read it, as in a workspace on the same home Volume: a path under /workspace is looked for under your home Volume too. Anywhere else it returns run.failure_message, and an empty string when there is nothing to show yet.

Python
run = get_job("tj-abc12")
if run.phase == "Failed":
    print(run.failure_reason, run.log_path)
    print(run.logs())

Stop or delete a run

run.stop() ends the job and releases its GPUs. The run's record stays, with the phase Stopped, and so does everything the job wrote to your Volumes. A stopped run can't start again. Submit a new one instead, and if the run saved checkpoints, resume from one. run.cancel() does the same as stop(). Before SDK 0.11.0, cancel() deleted the run.

run.delete() ends the job if it still runs and removes its record, so get_job no longer finds it. Files on your Volumes, the log included, stay.

Stopping or deleting a run that is already gone is not an error. Stopping one that has already ended raises ConflictError.

You can stop and delete your own runs. Org admins can stop and delete any run in the org.

Reading results back

run.metrics(), run.params() and run.list_artifacts() read the job's MLflow run, so you can pull results into the notebook without opening the MLflow UI. They only read.

Python
run.wait()

print(run.metrics())        # {'loss': 0.21, 'accuracy': 0.94}
print(run.params())         # {'lr': '0.001', 'epochs': '10'}

for a in run.list_artifacts():
    print(a.path, a.size if not a.is_dir else "(dir)")

list_artifacts() returns RunArtifact entries with path, is_dir and size (None for a directory). RunArtifact is the SDK's own type, so it doesn't change when MLflow does.

These need the mlflow extra (pip install 'privatemind[mlflow]') and a tracking URI. The SDK takes the URI from the tracking_uri= argument, then from PRIVATEMIND_MLFLOW_URI, and raises ConfigError if neither is set. Until a job starts tracking, run.mlflow_run_id is None and all three raise PrivatemindError, so wait for the run or check the id first.

For anything else, mlflow_client() returns a real MlflowClient pointed at the same URI:

Python
from privatemind import mlflow_client

client = mlflow_client()                  # or mlflow_client(tracking_uri=...)
history = client.get_metric_history(run.mlflow_run_id, "loss")

Completion webhooks

The platform can POST to a URL of yours when a job succeeds or fails, so you don't have to poll with run.wait(). Pass webhook_url when you submit, and webhook_secret to sign the callback:

Python
run = submit_training(
    entrypoint="python train.py",
    image="<training-image>",
    workers=1,
    target_cluster="<your-gpu-cluster>",
    webhook_url="https://hooks.acme.com/pmind/training",
    webhook_secret="whsec_...",   # optional; enables signing
)

webhook_url must be https. The SDK rejects anything else at submit, localhost included. Your org admin must also register the host as an approved egress (outbound) destination, or the submit is rejected (see the prerequisite below).

The platform sends one POST per job. Delivery is at least once, so a retry can send the same POST twice: drop duplicates by delivery_id.

JSON
{
  "job_id": "tj-abc12",
  "name": "tj-abc12",
  "delivery_id": "9f1c...",
  "status": "Succeeded",
  "started_at": "2026-01-02T10:00:00Z",
  "finished_at": "2026-01-02T10:14:30Z"
}

status is the terminal phase, Succeeded or Failed. Treat two fields as optional: a Failed body also carries an error string, and mlflow_run_id appears only once the run has one.

Verifying the signature

When you set webhook_secret, every delivery carries Standard Webhooks headers:

  • webhook-id: the delivery_id (the same across retries of one delivery).
  • webhook-timestamp: unix seconds.
  • webhook-signature: v1,<base64 HMAC-SHA256(secret, "{webhook-id}.{webhook-timestamp}.{raw-body}")>.

Verify with any Standard Webhooks library, or directly:

Python
import base64, hashlib, hmac

def verify(secret, headers, raw_body):
    signed = f"{headers['webhook-id']}.{headers['webhook-timestamp']}.{raw_body}"
    expected = base64.b64encode(
        hmac.new(secret.encode(), signed.encode(), hashlib.sha256).digest()
    ).decode()
    got = headers["webhook-signature"].removeprefix("v1,")
    return hmac.compare_digest(expected, got)

Verify over the raw request body. Parsing the JSON and dumping it again changes the bytes and breaks the signature. To guard against replays, reject deliveries whose webhook-timestamp is far from the current time.

Warning

Never pass a webhook_secret or webhook_url with leading or trailing whitespace. The server strips the secret before signing. If your receiver keeps a stray newline, it checks against a different value and every signature fails, with no error. An all-whitespace secret strips to empty, and the submit is rejected with 400. A secret read from a file usually ends in a newline: strip it with Path("webhook.secret").read_text().strip(). The SDK rejects padded values at submit with ValidationError.

Checking delivery from the SDK

run.webhook_status shows where a delivery stands, as the server reports it:

Value Meaning
not_configured no webhook_url was set
disabled a webhook is configured but delivery is off in this environment
pending queued or mid-retry
blocked_egress the host is not an approved egress destination for your org; nothing is delivered until an admin registers it
delivered the callback was accepted (2xx)
failed retries exhausted
unknown the server could not read the delivery state; check again shortly

The value comes from the last fetched status, so call refresh() first, or use get_job, which fetches:

Python
from privatemind import get_job

print(get_job("tj-abc12").webhook_status)   # e.g. "delivered"

It is None only if the server leaves the field out. Treat any value you don't recognise as unknown.

Prerequisite: the host must be allowlisted

Webhook deliveries leave PrivateMind only through your org's egress proxy, which reaches only the hosts your org has registered. So your org admin registers each webhook host once as an egress destination, and every later job can use it. A submit that names an unregistered host is rejected outright with 400, and nothing is queued. blocked_egress is the other case: the host was registered when you submitted but removed before the job finished. The delivery then waits until the host is registered again.

Promote a notebook function

@pm.train takes a function you've tested in the notebook and runs it as a training job tracked in MLflow, without leaving Python.

Python
import privatemind as pm

@pm.train(workers=1, gpus_per_worker=1)
def train(lr=3e-4, epochs=10):
    import mlflow, torch       # re-resolved on the worker, must be in the image
    assert torch.cuda.is_available()
    ...

train(lr=1e-3)                 # runs locally in the notebook (validate)
run = train.promote(lr=1e-3)   # packages + submits to the GPU -> Run
run.wait(); print(run.mlflow_run_id)
  • Calling the function, or train.local(...), runs it in the notebook.
  • train.promote(...) packages the function with its arguments, copies it to your home Volume and submits it through submit_training. It returns a Run.
  • train.with_options(gpus_per_worker=4) returns a copy with changed settings, so you don't have to decorate the function again.

The decorator takes the same arguments as submit_training, except entrypoint, namespace and the two webhook fields. It adds two of its own: experiment= groups the MLflow runs under a named experiment, and home_volume= sets the Volume that carries your code.

In a workspace, target_cluster and home_volume default from the workspace, and image comes from the platform's allowlist, so you usually pass only workers=1 and gpus_per_worker. Outside a workspace the image is found the same way, but target_cluster and home_volume have no default: pass both, or promote() raises ConfigError.

Finding the image needs SDK 0.4.0 or newer. Older versions look for a PRIVATEMIND_TRAINING_IMAGE variable the platform doesn't set, and raise ConfigError naming it. Pass image= yourself, or upgrade with pip install -U privatemind.

Three rules make a function promotable:

  1. Take all inputs as arguments. The function and its arguments travel to the job as one serialised payload. A notebook global in the function, such as a DataFrame or a loaded model, either fails with ValidationError (the function "isn't self-contained") or bloats the payload. The SDK warns when the payload passes about 5 MB. Read data from mounted Volumes, and write outputs to a Volume or to MLflow.
  2. Import inside the function. Imports run again on the worker against the training image, not the notebook's installed packages, so import only what the image provides.
  3. GPU code runs on a GPU worker. With gpus_per_worker above 0, promote() runs your function on a GPU worker, so plain torch code that uses CUDA works as written.

If your org has MLflow set up, every promote is tracked unless you pass mlflow=False:

  • The run is named after the job, which is named after your function unless you pass name=.
  • CPU, GPU and memory system metrics are recorded.
  • Anything you log inside the function, such as metrics, artifacts or mlflow.pytorch.log_model(...), goes to that run.

If the job can't reach MLflow when it starts, the function runs untracked and the job's log says so.

To pull the results into the notebook afterwards, see Reading results back.

Mounting data

volumes mounts existing Volumes into every worker. Each entry is a mapping with these keys:

  • volume: the Volume's name.
  • mount_path: an absolute path, unique within the job.
  • read_only: optional, default False. It must be a real boolean, not a string.
Python
run = submit_training(
    entrypoint="python train.py --data /data --out /artifacts",
    image="<training-image>",
    workers=1,
    target_cluster="<your-gpu-cluster>",
    gpus_per_worker=1,
    volumes=[
        {"volume": "datasets", "mount_path": "/data", "read_only": True},
        {"volume": "checkpoints", "mount_path": "/artifacts"},
    ],
)

The SDK rejects relative or duplicate mount paths before submitting. With @pm.train, your home Volume is always mounted as well, because it carries the promoted code.

Mounting registry weights

weights mounts versions from your org's registry read-only into the job, up to 8, each given as <name>:<version>. A version goes in /home/privatemind/models/<folder>, the folder named after the last part of the model's name. To choose the folder, pass a mapping with the ref and a mount_path, as here:

Python
run = submit_training(
    entrypoint="python train.py --base /home/privatemind/models/base",
    image="<training-image>",
    workers=1,
    target_cluster="<your-gpu-cluster>",
    gpus_per_worker=8,
    weights=[{"ref": "credit-ft:2026-11-16", "mount_path": "/home/privatemind/models/base"}],
)

The folder takes lowercase letters, digits and dashes, up to 63 characters. This mapping differs from the one workspace jobs take, which names the folder with name: a training job ignores name. The platform pins each version to its digest when you submit.

Explicit client

For a longer session, or to manage configuration yourself, use a Client as a context manager. Both URLs default to the production platform, so pass them only to point somewhere else:

Python
from privatemind import Client

with Client(
    base_url="https://api.privatemind.com",   # platform API (images, batch)
    app_url="https://privatemind.com",        # app backend (training jobs)
    token="PMIND...:abcdef...",
) as pm:
    run = pm.submit_training(
        entrypoint="python train.py",
        image="<training-image>",
        workers=1,
        target_cluster="<your-gpu-cluster>",
    )
    for r in pm.list_jobs():
        print(r.name, r.phase)

The module-level functions (submit_training, list_jobs, get_job) share one client per process, created from the environment on first use. reconfigure(base_url=..., app_url=..., token=...) replaces it, which helps in a notebook when credentials change. Each function also takes client= to send a single call through a client of your own.

Each error has its own type (AuthError, ConfigError, ValidationError, ForbiddenError, NotFoundError, ConflictError, RateLimitError, ServerError, UnknownPhaseError), all subclasses of PrivatemindError. The SDK overview covers them in full.

From a shell

The privatemind command that comes with the SDK lists, reads and stops training runs. It authenticates the same way as the SDK: in a workspace it is ready to use, and elsewhere it reads PRIVATEMIND_TOKEN.

Command What it does
privatemind jobs ls Your training runs, with phase, start and end times, and failure reason.
privatemind jobs get <name> One run: its phase, times, MLflow run, failure reason, log path, link and failure message.
privatemind jobs logs <name> The run's log, as run.logs() returns it.
privatemind jobs stop <name> Stop the run and keep its record.

Where next

  • Compute & training: what the platform offers and how it fits together.
  • Fine-tuning: pm.finetune does all of this in one call, for SFT, DPO, KTO and GRPO.
  • Evaluate and publish: score a fine-tuned model, push it to your registry and serve it.
  • SDK overview: configuration, authentication, and the full error catalog.
  • Workspaces: the GPU notebook environment where the SDK is set up for you.
  • API keys: create the key the SDK authenticates with outside a workspace.