# Training jobs Submit and manage training jobs from any notebook, script or shell. > [!NOTE] > The SDK uses your org's GPU allocation. If your org doesn't have one, training calls return an error. See [Compute & training](https://docs.privatemind.com/training.html). The [PrivateMind Python SDK](https://docs.privatemind.com/sdk.html) submits and manages training jobs, with the same calls from a notebook, a script or CI. It comes installed in every [workspace](https://docs.privatemind.com/workspaces.html), with your token, the API URLs and your GPU cluster name already set as environment variables. There, you `import privatemind` and call `submit_training`. Failure reasons, logs, stopping a run, registry weights and the `privatemind jobs` command need SDK 0.11.0 or later: `pip install -U "privatemind>=0.11"`. ## Authentication In a workspace you can skip this section, because credentials and URLs are already set. Outside a workspace, on a laptop or in CI, the SDK looks for a token in this order: 1. `token=` passed to `Client(...)` 2. the `PRIVATEMIND_TOKEN` environment variable 3. `~/.privatemind/auth` (must be mode `0600`, owned by you, and not a symlink) The token is a [PrivateMind API key](https://docs.privatemind.com/api-keys.html). The SDK talks to two services, and training uses the second: - **Platform API**: `base_url=` or `PRIVATEMIND_URL`, default `https://api.privatemind.com`. Serves [image generation](https://docs.privatemind.com/sdk-images.html) and [batch](https://docs.privatemind.com/sdk-batch.html). - **App backend**: `app_url=` or `PRIVATEMIND_APP_URL`, default `https://privatemind.com`. Serves the training methods (`submit_training`, `list_jobs`, `get_job`). Both must use `https://`. The [SDK overview](https://docs.privatemind.com/sdk.html) covers the rest of the configuration. ## Submit a job `submit_training` starts a job and returns a [`Run`](https://docs.privatemind.com/training-sdk.html#the-run-handle) handle. All its arguments are keyword-only. ```python from privatemind import submit_training run = submit_training( entrypoint="python train.py --epochs 10", image="", workers=1, target_cluster="", gpus_per_worker=1, ) print(run) # Run(name='tj-abc12', phase='Pending') print(run.url) # link to the run in the PrivateMind UI run.wait() # block until the job is terminal print(run.phase, run.mlflow_run_id) ``` `image` must be one of the platform's own training images. The platform refuses any other image, your own included. There are two: the default PyTorch image, and an image that adds vLLM, which [GRPO](https://docs.privatemind.com/sdk-grpo.html) and [evaluation](https://docs.privatemind.com/sdk-evaluate.html) run on. On SDK 0.4.0 or newer, `Client().list_training_images()` lists every offered version of both, with the default first. Pass an entry's `value` unchanged, digest included. | Parameter | Default | Purpose | |---|---|---| | `entrypoint` | required | Shell command the job runs on its Ray head, e.g. `python train.py --epochs 10`. | | `image` | required | Container image the workers run, from the platform's list (see above). | | `workers` | required | Number of Ray workers. Training is supported on one node: use `1`, and set the GPUs with `gpus_per_worker`. The SDK refuses more than 64, or 16 for a job with GPUs. | | `target_cluster` | required | Your org's GPU cluster name (from your dashboard or account team). | | `gpus_per_worker` | `0` | GPUs per worker, 0 to 8. `0` is CPU-only. | | `cpu_per_worker` | none | CPU request per worker: an int or a Kubernetes quantity string (`"2"`, `"500m"`). | | `memory_per_worker` | none | Memory request per worker, same format (`"8Gi"`). | | `env` | none | Environment variables for the workers. Keys starting with `MLFLOW_` are reserved by the platform and rejected. | | `mlflow` | `True` | Track the run in MLflow, which fills in `run.mlflow_run_id`. If your org has no MLflow set up, pass `False` or the job is refused. | | `ray_version` | platform default | Pin the Ray version the job runs. | | `ttl_seconds_after_finished` | `600` | Seconds the job's Ray cluster stays up after the job ends, whether it succeeded or failed, 0 to 86400 (one day). The platform then removes the cluster. The run's record stays until you delete it. | | `name` | assigned | Job name: lowercase letters, digits and hyphens, up to 40 characters, starting and ending with a letter or digit. Omit it and the platform assigns one. | | `namespace` | assigned | Advanced override. Omit it in normal use. | | `volumes` | none | Volumes mounted into every worker. See [Mounting data](https://docs.privatemind.com/training-sdk.html#mounting-data). | | `weights` | none | Versions from your org's [registry](https://docs.privatemind.com/registry.html) to mount read-only. See [Mounting registry weights](https://docs.privatemind.com/training-sdk.html#mounting-registry-weights). | | `gpu_placements` | auto-derived | Pin exact GPUs. Omit it and the platform picks them. See [GPUs](https://docs.privatemind.com/training-sdk.html#gpus). | | `webhook_url` | none | HTTPS URL the platform POSTs to when the job finishes. See [Completion webhooks](https://docs.privatemind.com/training-sdk.html#completion-webhooks). | | `webhook_secret` | none | Signs each webhook delivery. Requires `webhook_url`. | | `client` | shared client | An explicit [`Client`](https://docs.privatemind.com/training-sdk.html#explicit-client) to submit through. | The SDK checks arguments before it sends anything. A bad value, such as `workers=0`, a malformed `name` or a reserved `env` key, raises `ValidationError`. A run's RAM and vCPU count against your personal compute allowance, which workspaces, inference deployments and jobs also use. The total covers every worker plus the job's Ray head, the node that runs your entrypoint. See [Personal compute quotas](https://docs.privatemind.com/training.html#personal-compute-quotas). > [!TIP] > In a workspace, `os.environ["PRIVATEMIND_TARGET_CLUSTER"]` holds your cluster name. The [`@pm.train`](https://docs.privatemind.com/training-sdk.html#promote-a-notebook-function) decorator fills in `target_cluster` and `image` for you. ### GPUs Usually you set `gpus_per_worker` and leave `gpu_placements` unset. The platform then pins the job to free GPUs your org owns, so you don't need to know which ones you have. To pin exact devices, pass one placement per worker. Each is a mapping with a `host` and exactly `gpus_per_worker` GPU `indices` (integers 0 to 7, no duplicates): ```python gpu_placements=[{"host": "", "indices": [0, 1]}] # workers=1, gpus_per_worker=2 ``` The SDK checks the shape when you submit: `gpu_placements` needs `gpus_per_worker` above 0, and the list length must equal `workers`. The platform checks ownership, index conflicts and quota, and rejects a job that asks for GPUs it cannot have. Your entrypoint runs on the job's Ray head, which has no GPU. The GPUs are on the workers, so your script reaches them through Ray (Ray Train, or `ray.remote(num_gpus=...)`) as in any Ray program. The [`@pm.train`](https://docs.privatemind.com/training-sdk.html#promote-a-notebook-function) decorator does this for you. ## The Run handle A `Run` is a live handle to the job. Its `name`, `namespace` and `url` never change. Its status is a snapshot. Status properties such as `run.phase` return the last status fetched, and fetch it the first time you read one. Call `run.refresh()` to fetch it again. | Member | What it gives you | |---|---| | `run.phase` | Lifecycle state: `Pending`, `Queued` or `Running` while the job is live, then `Succeeded`, `Failed`, `Stopped` or `Terminating`. | | `run.url` | Link to the run in the PrivateMind UI. | | `run.failure_reason` | Why a failed run failed, as one word, else `None`. See [When a run fails](https://docs.privatemind.com/training-sdk.html#when-a-run-fails). | | `run.failure_message` | The last 8 KiB of the failure detail, else `None`. | | `run.log_path` | Where the job writes its full log, as a path inside the job. `None` when the job mounts no writable Volume. | | `run.logs()` | The job's driver log, or the failure message where the log can't be read. See [Logs](https://docs.privatemind.com/training-sdk.html#logs). | | `run.mlflow_run_id` | MLflow run id once tracking has started, else `None`. | | `run.ray_job_name` | Underlying Ray submission id once the job is admitted, else `None`. | | `run.start_time`, `run.end_time` | RFC 3339 timestamps, `None` until the job starts or finishes. | | `run.webhook_status` | Completion-webhook delivery state. See [Checking delivery from the SDK](https://docs.privatemind.com/training-sdk.html#checking-delivery-from-the-sdk). | | `run.refresh()` | Fetch the status again. Returns the run. | | `run.wait(timeout=None, poll_interval=5.0)` | Block until the job stops running. Returns the run. | | `run.watch(timeout=None, poll_interval=5.0)` | Block like `wait()`, showing a live status table in a notebook. Returns the run. | | `run.stop()` | End the job and keep its record. See [Stop or delete a run](https://docs.privatemind.com/training-sdk.html#stop-or-delete-a-run). | | `run.cancel()` | The same as `stop()`. | | `run.delete()` | End the job if it still runs, and remove its record. | | `run.publish(name, version=None)` | Push a fine-tune's result to your org's registry and return `:`. See [Evaluate and publish](https://docs.privatemind.com/sdk-evaluate.html). | | `run.metrics()` | Latest metric values from this run's MLflow run, as `{name: float}`. | | `run.params()` | Params logged to this run's MLflow run, as `{name: str}`. | | `run.list_artifacts(path=None)` | Artifact entries under the run, optionally rooted at `path`. | A job reports one of seven phases. The SDK groups them into two sets, both importable from `privatemind`: | Phase | Set | Meaning | |---|---|---| | `Pending` | `ACTIVE_PHASES` | the job was accepted; the platform has not started on it yet | | `Queued` | `ACTIVE_PHASES` | the workers exist and are waiting for GPUs to come free | | `Running` | `ACTIVE_PHASES` | the workers are up and your entrypoint is executing | | `Succeeded` | `TERMINAL_PHASES` | the job exited cleanly | | `Failed` | `TERMINAL_PHASES` | the job exited non-zero, or the platform could not run it. `run.failure_reason` says which | | `Stopped` | `TERMINAL_PHASES` | you or an admin stopped the job | | `Terminating` | `TERMINAL_PHASES` | tear-down is under way. You are unlikely to see it: the platform doesn't report it, and a deleted job stops answering. The SDK counts it as terminal so a job that reaches it ends the wait | `wait()` polls while the phase is in `ACTIVE_PHASES` and returns as soon as it isn't. - `poll_interval` is the first gap between polls, in seconds. It doubles each poll, up to 30 seconds, or up to `poll_interval` if you set it higher. - `timeout=None` waits forever. If the job is still running when a timeout expires, `wait()` raises `TimeoutError`. The job keeps running until it finishes or you cancel it. A phase in neither set raises `UnknownPhaseError`, which carries `.run_name` and `.phase`. It means the platform reports a phase your installed SDK doesn't know. The SDK then can't tell whether the job has finished, so it won't say it has. Upgrade `privatemind`, or read `run.phase` and decide yourself. SDK versions before 0.11.0 don't know `Stopped`, so waiting on a stopped run raises this error there. `wait(on_unknown_phase="return")` returns with a `UserWarning` and doesn't raise, but keep the default. If your notebook runs `warnings.filterwarnings("ignore")`, that return is silent. `run.metrics()` then gives you half a training run's numbers with nothing on screen to say so. ```python try: run.wait(timeout=3600) except TimeoutError: run.stop() if run.phase == "Succeeded": print(run.mlflow_run_id, run.end_time) elif run.phase == "Failed": print(run.failure_reason) print(run.logs()) ``` You can list or look up jobs without the original handle: - `list_jobs()` returns a `RunList` of handles with their status filled in: a `Run` for each training run and a `Job` for each [workspace job](https://docs.privatemind.com/sdk-jobs.html). Each item's `kind` is `"training"` or `"job"`. On SDK versions before 0.5.0 it always returns an empty list, so upgrade first. - `phase=` keeps only jobs in that phase, for both kinds. The SDK does the filtering, on an exact match that ignores case. - `list_training_jobs()` returns only the training runs. - `get_job(name)` looks up a training run, and `get_workspace_job(name)` a workspace job. ```python from privatemind import list_jobs, get_job for r in list_jobs(): # training runs and workspace jobs you may see print(r.kind, r.name, r.phase) running = list_jobs(phase="Running") # or "running"; matched case-insensitively get_job("tj-abc12").stop() ``` > [!NOTE] > A `Run` holds an authenticated client, so it can't be pickled or deep-copied. To act on a job from another process, pass the job name and get a new handle there with `get_job(name)`. ### Watching progress `wait()` prints nothing by default. In a script or CI log, pass `progress=True` to print a timestamped line per phase change and a last line with the total time: ```python run.wait(progress=True) # [12:00:01] run tj-abc12: Pending # [12:00:06] run tj-abc12: Pending -> Running # [12:11:43] run tj-abc12: Running -> Succeeded # [12:11:43] run tj-abc12: Succeeded after 702.0s (terminal) ``` To handle changes yourself, pass a function as `on_change=`. `wait()` calls it with the new phase on every change it sees, including the move to a terminal phase. The phase `wait()` sees when it starts doesn't count as a change. So if the job is already `Running`, your function is called once, on `Succeeded`. An exception in your function propagates out of `wait()`: ```python run.wait(on_change=lambda phase: print("now:", phase)) ``` In a notebook, `run.watch()` shows a status table that updates on each change. It takes the same `timeout`, `poll_interval` and `on_unknown_phase` arguments as `wait()`. In a plain script, or a kernel without IPython, it falls back to `wait(progress=True)`: ```python run.watch() ``` A `Run` or `RunList` also shows as a table when it is the last expression in a cell. The table makes no request. It shows the status the handle last fetched, or `?` in every status column if it has never fetched one: ```python run # one run, as a field and value table list_jobs(phase="Running") # one row per run ``` Slicing a `RunList`, or adding two together, gives another `RunList`, which keeps the table. `sorted()` and the other builtins that copy a list return a plain `list`, which doesn't. ## When a run fails A failed run carries a reason and a message. `run.failure_reason` is one of these: | Reason | What happened | |---|---| | `JobFailed` | Your entrypoint exited non-zero, or Ray ended the job. The message holds the end of its output, where the error usually is. | | `GPUUnavailable` | A GPU the job was given could not be prepared for 5 minutes, so the platform failed the job. The message is the GPU driver's. Submit again on other GPUs with [`gpu_placements`](https://docs.privatemind.com/training-sdk.html#gpus), and tell your account team which GPU it was. | | `ValidationFailed` | Ray refused the job before it started. The message says which setting. | | `DeploymentFailed` | The job's Ray cluster could not be started. | `run.failure_message` keeps the last 8 KiB of the failure detail. For a long traceback that is the end, with the exception. For the whole output, read the log. ### Logs The platform saves everything your entrypoint prints, stdout and stderr, to a file on the first Volume the job mounts writable: ``` /.privatemind/training//driver.log ``` `run.log_path` gives that path as the job sees it. The file is on your Volume, so it outlives the job and its Ray cluster. A job that mounts no writable Volume has no log file, and `run.log_path` is `None`. `pm.finetune` and `@pm.train` mount your home Volume first. In a workspace on that Volume, the log of job `tj-abc12` is at `~/.privatemind/training/tj-abc12/driver.log`. `run.logs()` returns the file's contents when the SDK can read it, as in a workspace on the same home Volume: a path under `/workspace` is looked for under your home Volume too. Anywhere else it returns `run.failure_message`, and an empty string when there is nothing to show yet. ```python run = get_job("tj-abc12") if run.phase == "Failed": print(run.failure_reason, run.log_path) print(run.logs()) ``` ## Stop or delete a run `run.stop()` ends the job and releases its GPUs. The run's record stays, with the phase `Stopped`, and so does everything the job wrote to your Volumes. A stopped run can't start again. Submit a new one instead, and if the run saved checkpoints, [resume from one](https://docs.privatemind.com/sdk-finetune.html#checkpoints-and-resume). `run.cancel()` does the same as `stop()`. Before SDK 0.11.0, `cancel()` deleted the run. `run.delete()` ends the job if it still runs and removes its record, so `get_job` no longer finds it. Files on your Volumes, the log included, stay. Stopping or deleting a run that is already gone is not an error. Stopping one that has already ended raises `ConflictError`. You can stop and delete your own runs. Org admins can stop and delete any run in the org. ## Reading results back `run.metrics()`, `run.params()` and `run.list_artifacts()` read the job's MLflow run, so you can pull results into the notebook without opening the MLflow UI. They only read. ```python run.wait() print(run.metrics()) # {'loss': 0.21, 'accuracy': 0.94} print(run.params()) # {'lr': '0.001', 'epochs': '10'} for a in run.list_artifacts(): print(a.path, a.size if not a.is_dir else "(dir)") ``` `list_artifacts()` returns `RunArtifact` entries with `path`, `is_dir` and `size` (`None` for a directory). `RunArtifact` is the SDK's own type, so it doesn't change when MLflow does. These need the `mlflow` extra (`pip install 'privatemind[mlflow]'`) and a tracking URI. The SDK takes the URI from the `tracking_uri=` argument, then from `PRIVATEMIND_MLFLOW_URI`, and raises `ConfigError` if neither is set. Until a job starts tracking, `run.mlflow_run_id` is `None` and all three raise `PrivatemindError`, so wait for the run or check the id first. For anything else, `mlflow_client()` returns a real `MlflowClient` pointed at the same URI: ```python from privatemind import mlflow_client client = mlflow_client() # or mlflow_client(tracking_uri=...) history = client.get_metric_history(run.mlflow_run_id, "loss") ``` ## Completion webhooks The platform can POST to a URL of yours when a job succeeds or fails, so you don't have to poll with `run.wait()`. Pass `webhook_url` when you submit, and `webhook_secret` to sign the callback: ```python run = submit_training( entrypoint="python train.py", image="", workers=1, target_cluster="", webhook_url="https://hooks.acme.com/pmind/training", webhook_secret="whsec_...", # optional; enables signing ) ``` `webhook_url` must be `https`. The SDK rejects anything else at submit, localhost included. Your org admin must also register the host as an approved egress (outbound) destination, or the submit is rejected (see the prerequisite below). The platform sends one POST per job. Delivery is at least once, so a retry can send the same POST twice: drop duplicates by `delivery_id`. ```json { "job_id": "tj-abc12", "name": "tj-abc12", "delivery_id": "9f1c...", "status": "Succeeded", "started_at": "2026-01-02T10:00:00Z", "finished_at": "2026-01-02T10:14:30Z" } ``` `status` is the terminal phase, `Succeeded` or `Failed`. Treat two fields as optional: a `Failed` body also carries an `error` string, and `mlflow_run_id` appears only once the run has one. ### Verifying the signature When you set `webhook_secret`, every delivery carries [Standard Webhooks](https://www.standardwebhooks.com/) headers: - `webhook-id`: the `delivery_id` (the same across retries of one delivery). - `webhook-timestamp`: unix seconds. - `webhook-signature`: `v1,`. Verify with any Standard Webhooks library, or directly: ```python import base64, hashlib, hmac def verify(secret, headers, raw_body): signed = f"{headers['webhook-id']}.{headers['webhook-timestamp']}.{raw_body}" expected = base64.b64encode( hmac.new(secret.encode(), signed.encode(), hashlib.sha256).digest() ).decode() got = headers["webhook-signature"].removeprefix("v1,") return hmac.compare_digest(expected, got) ``` Verify over the **raw request body**. Parsing the JSON and dumping it again changes the bytes and breaks the signature. To guard against replays, reject deliveries whose `webhook-timestamp` is far from the current time. > [!WARNING] > Never pass a `webhook_secret` or `webhook_url` with leading or trailing whitespace. The server strips the secret before signing. If your receiver keeps a stray newline, it checks against a different value and every signature fails, with no error. An all-whitespace secret strips to empty, and the submit is rejected with `400`. A secret read from a file usually ends in a newline: strip it with `Path("webhook.secret").read_text().strip()`. The SDK rejects padded values at submit with `ValidationError`. ### Checking delivery from the SDK `run.webhook_status` shows where a delivery stands, as the server reports it: | Value | Meaning | |---|---| | `not_configured` | no `webhook_url` was set | | `disabled` | a webhook is configured but delivery is off in this environment | | `pending` | queued or mid-retry | | `blocked_egress` | the host is not an approved egress destination for your org; nothing is delivered until an admin registers it | | `delivered` | the callback was accepted (2xx) | | `failed` | retries exhausted | | `unknown` | the server could not read the delivery state; check again shortly | The value comes from the last fetched status, so call `refresh()` first, or use `get_job`, which fetches: ```python from privatemind import get_job print(get_job("tj-abc12").webhook_status) # e.g. "delivered" ``` It is `None` only if the server leaves the field out. Treat any value you don't recognise as `unknown`. ### Prerequisite: the host must be allowlisted Webhook deliveries leave PrivateMind only through your org's egress proxy, which reaches only the hosts your org has registered. So your **org admin registers each webhook host once** as an egress destination, and every later job can use it. A submit that names an unregistered host is rejected outright with `400`, and nothing is queued. `blocked_egress` is the other case: the host was registered when you submitted but removed before the job finished. The delivery then waits until the host is registered again. ## Promote a notebook function `@pm.train` takes a function you've tested in the notebook and runs it as a training job tracked in MLflow, without leaving Python. ```python import privatemind as pm @pm.train(workers=1, gpus_per_worker=1) def train(lr=3e-4, epochs=10): import mlflow, torch # re-resolved on the worker, must be in the image assert torch.cuda.is_available() ... train(lr=1e-3) # runs locally in the notebook (validate) run = train.promote(lr=1e-3) # packages + submits to the GPU -> Run run.wait(); print(run.mlflow_run_id) ``` - Calling the function, or `train.local(...)`, runs it in the notebook. - `train.promote(...)` packages the function with its arguments, copies it to your home Volume and submits it through `submit_training`. It returns a [`Run`](https://docs.privatemind.com/training-sdk.html#the-run-handle). - `train.with_options(gpus_per_worker=4)` returns a copy with changed settings, so you don't have to decorate the function again. The decorator takes the same arguments as `submit_training`, except `entrypoint`, `namespace` and the two webhook fields. It adds two of its own: `experiment=` groups the MLflow runs under a named experiment, and `home_volume=` sets the Volume that carries your code. In a workspace, `target_cluster` and `home_volume` default from the workspace, and `image` comes from the platform's allowlist, so you usually pass only `workers=1` and `gpus_per_worker`. Outside a workspace the image is found the same way, but `target_cluster` and `home_volume` have no default: pass both, or `promote()` raises `ConfigError`. Finding the image needs SDK 0.4.0 or newer. Older versions look for a `PRIVATEMIND_TRAINING_IMAGE` variable the platform doesn't set, and raise `ConfigError` naming it. Pass `image=` yourself, or upgrade with `pip install -U privatemind`. Three rules make a function promotable: 1. **Take all inputs as arguments.** The function and its arguments travel to the job as one serialised payload. A notebook global in the function, such as a DataFrame or a loaded model, either fails with `ValidationError` (the function "isn't self-contained") or bloats the payload. The SDK warns when the payload passes about 5 MB. Read data from mounted Volumes, and write outputs to a Volume or to MLflow. 2. **Import inside the function.** Imports run again on the worker against the training image, not the notebook's installed packages, so import only what the image provides. 3. **GPU code runs on a GPU worker.** With `gpus_per_worker` above 0, `promote()` runs your function on a GPU worker, so plain `torch` code that uses CUDA works as written. If your org has MLflow set up, every promote is tracked unless you pass `mlflow=False`: - The run is named after the job, which is named after your function unless you pass `name=`. - CPU, GPU and memory system metrics are recorded. - Anything you log inside the function, such as metrics, artifacts or `mlflow.pytorch.log_model(...)`, goes to that run. If the job can't reach MLflow when it starts, the function runs untracked and the job's log says so. To pull the results into the notebook afterwards, see [Reading results back](https://docs.privatemind.com/training-sdk.html#reading-results-back). ## Mounting data `volumes` mounts existing [Volumes](https://docs.privatemind.com/workspaces.html) into every worker. Each entry is a mapping with these keys: - `volume`: the Volume's name. - `mount_path`: an absolute path, unique within the job. - `read_only`: optional, default `False`. It must be a real boolean, not a string. ```python run = submit_training( entrypoint="python train.py --data /data --out /artifacts", image="", workers=1, target_cluster="", gpus_per_worker=1, volumes=[ {"volume": "datasets", "mount_path": "/data", "read_only": True}, {"volume": "checkpoints", "mount_path": "/artifacts"}, ], ) ``` The SDK rejects relative or duplicate mount paths before submitting. With `@pm.train`, your home Volume is always mounted as well, because it carries the promoted code. ### Mounting registry weights `weights` mounts versions from your org's [registry](https://docs.privatemind.com/registry.html) read-only into the job, up to 8, each given as `:`. A version goes in `/home/privatemind/models/`, the folder named after the last part of the model's name. To choose the folder, pass a mapping with the `ref` and a `mount_path`, as here: ```python run = submit_training( entrypoint="python train.py --base /home/privatemind/models/base", image="", workers=1, target_cluster="", gpus_per_worker=8, weights=[{"ref": "credit-ft:2026-11-16", "mount_path": "/home/privatemind/models/base"}], ) ``` The folder takes lowercase letters, digits and dashes, up to 63 characters. This mapping differs from the one [workspace jobs](https://docs.privatemind.com/sdk-weights.html#use-a-version-in-a-job-or-a-model-server) take, which names the folder with `name`: a training job ignores `name`. The platform pins each version to its digest when you submit. ## Explicit client For a longer session, or to manage configuration yourself, use a `Client` as a context manager. Both URLs default to the production platform, so pass them only to point somewhere else: ```python from privatemind import Client with Client( base_url="https://api.privatemind.com", # platform API (images, batch) app_url="https://privatemind.com", # app backend (training jobs) token="PMIND...:abcdef...", ) as pm: run = pm.submit_training( entrypoint="python train.py", image="", workers=1, target_cluster="", ) for r in pm.list_jobs(): print(r.name, r.phase) ``` The module-level functions (`submit_training`, `list_jobs`, `get_job`) share one client per process, created from the environment on first use. `reconfigure(base_url=..., app_url=..., token=...)` replaces it, which helps in a notebook when credentials change. Each function also takes `client=` to send a single call through a client of your own. Each error has its own type (`AuthError`, `ConfigError`, `ValidationError`, `ForbiddenError`, `NotFoundError`, `ConflictError`, `RateLimitError`, `ServerError`, `UnknownPhaseError`), all subclasses of `PrivatemindError`. The [SDK overview](https://docs.privatemind.com/sdk.html) covers them in full. ## From a shell The `privatemind` command that comes with the SDK lists, reads and stops training runs. It authenticates the same way as the SDK: in a workspace it is ready to use, and elsewhere it reads `PRIVATEMIND_TOKEN`. | Command | What it does | |---|---| | `privatemind jobs ls` | Your training runs, with phase, start and end times, and failure reason. | | `privatemind jobs get ` | One run: its phase, times, MLflow run, failure reason, log path, link and failure message. | | `privatemind jobs logs ` | The run's log, as `run.logs()` returns it. | | `privatemind jobs stop ` | Stop the run and keep its record. | ## Where next - [Compute & training](https://docs.privatemind.com/training.html): what the platform offers and how it fits together. - [Fine-tuning](https://docs.privatemind.com/sdk-finetune.html): `pm.finetune` does all of this in one call, for SFT, DPO, KTO and GRPO. - [Evaluate and publish](https://docs.privatemind.com/sdk-evaluate.html): score a fine-tuned model, push it to your registry and serve it. - [SDK overview](https://docs.privatemind.com/sdk.html): configuration, authentication, and the full error catalog. - [Workspaces](https://docs.privatemind.com/workspaces.html): the GPU notebook environment where the SDK is set up for you. - [API keys](https://docs.privatemind.com/api-keys.html): create the key the SDK authenticates with outside a workspace.