Get API key

Workspace jobs

View as Markdown

Run one command in a workspace image on your home volume, without opening a workspace: from the Jobs tab, from the Python SDK, or on a schedule.

Note

Jobs run in your org's compute allocation. A CPU job needs only compute access; a GPU job holds whole cards your org owns and is refused at submit when none is free. See Compute & training.

What a workspace job is

A workspace job runs one shell command in the same environment as a workspace, without the notebook, editor and terminal. The platform starts the workspace template you choose, mounts your home volume, runs the command as you in your home directory, and stops when the command does. The command's exit code becomes the job's exit code, and the output is saved on the home volume.

Because the image and the volume are the same, your files are where a workspace would put them. A script at ~/projects/etl/run.py in a workspace is at the same path in a job, and whatever the job writes is there the next time you open a workspace on that volume.

Two things are different from typing the command into a workspace terminal:

  • There is no login shell. The command runs with the job's environment, and nothing in .bashrc, .zshrc or .profile is read. If the command needs a directory, a variable or an activated environment, put the cd, the export or the source in the command itself.
  • There is a deadline. Every job has a maximum duration: 24 hours unless you set one, and at most seven days. When it passes, the platform stops the job and it ends Failed with the reason DeadlineExceeded.

The command can be up to 4096 characters. Anything longer belongs in a script on the home volume, with the command calling the script.

When to use one

Use a workspace job for work that runs on its own, so you don't keep a workspace open for it. For example, a data preparation step, an evaluation over a checkpoint, or a script that rebuilds a dataset overnight. If it runs as a script from the workspace terminal, it runs as a job. A job can run on the home volume of a workspace that is open at the same time, and both see the same files live.

A training job is the other way to run work without a notebook. It runs as a Ray job on the platform's training images, with every run tracked in MLflow. A workspace job is one process, in the environment you already work in, with an exit code and a log. Use a workspace job for one command in your own environment, and a training job for a fine-tune, a Ray program or experiment tracking. A workspace job can also submit a training job: see Inside a job.

What happens when it runs

A job moves through four phases:

  • Pending: the platform is finding the job a place to run and pulling the template image. The maximum duration is already counting: it starts when the job starts trying to run, not when the command begins.
  • Running: the command is executing.
  • Succeeded: the command exited with code 0.
  • Failed: the command exited with a non-zero code, the platform stopped the job, or the job could not start. A failed job carries a reason and a message that say which.

The exit code is present only when the command actually exited, so a job stopped at its deadline has none. If no machine can take the job for a minute, it ends Failed with the reason Unschedulable and the scheduler's message. It doesn't wait for its deadline.

After a job ends, its record (phase, exit code, the tail of the output and the settings it ran with) stays for one day and is then removed. You can ask for a longer life at submit, up to seven days. The output file on the home volume stays as long as the volume does.

Seeing what happened

The job's output is written to the home volume at ~/.privatemind/jobs/<name>/stdout.log as it runs. Open a workspace on the same volume and you can read the file, or follow it with tail -f, while the job is still going.

When the job ends, the last 16 KiB of that output are recorded on the job itself. The Jobs tab shows that tail when you open a job, and logs() returns it in the SDK. While a job is running the tail is empty, so read the file. A job stopped at its deadline usually has no tail either, because the platform could not read it in time; the file on the home volume is the record then.

Volumes and who can see a job

A job mounts your home volume, and up to 16 more volumes by name, the way a workspace mounts them. The home volume must already exist; a job does not create one.

A job is private to you unless you submit it as shared, which makes it visible to the whole org, like a shared workspace. A job that mounts a personal volume stays personal whatever you ask. A volume that belongs to another member and is not shared with you can't be mounted at all, and the submit is refused (see Refused at submit).

GPUs

Ask for 0 to 8 whole cards. They are held for the job's whole life, from submit until it ends, whether or not the command is using them.

In the Jobs tab you choose a host and tick free cards on it that your org owns. From the SDK or an automation you can name the cards the same way with gpu_placements, or give a count and let the platform pick free cards your org owns on one machine. There is no queue: if no card is free, or a card you pinned is busy, the submit is refused right away.

Note

When the platform picks the cards, it chooses a machine by free cards and does not check its free memory. If the memory your template asks for by default does not fit on that machine, the job fails after about a minute with the reason Unschedulable and the scheduler's message (for example, insufficient memory). Set memory to a value that fits, or pin the cards on another machine with gpu_placements.

Refused at submit

The platform checks a job when you submit it and refuses with a message that says what is wrong:

  • the home volume does not exist;
  • a volume belongs to another member and is not shared with you (volume '<name>' is private to another member);
  • the template, or the version you asked for, is not one your org offers;
  • the name is already in use;
  • no card is free, or a card you pinned is busy (the message names the workload holding it);
  • your personal compute quota would be exceeded;
  • a shared job names a network profile that is restricted to some members.

From the Jobs tab

The Jobs tab, under Compute > Workspaces, lists the workspace jobs you may see, each with its phase, exit code and duration. Open a job for its output. Training runs are on the Training tab of Inference & Training.

Run a job asks for the same things submit_job takes: a template and version, the command, the home volume and any extra volumes, the cards, CPU and memory, a maximum duration, and whether the job is shared. The table under Submit a job says what each one means. When your organisation has Secrets on, the form also takes secrets, which the job reads as files.

On the jobs list:

  • Delete ends a running job and removes its record.
  • Run again, on a finished job of yours, opens Run a job filled in from it.
  • Schedule, on a job row of yours, opens Run a job set to On a schedule and filled in from that job, so a job you ran once by hand becomes a schedule without retyping. Any secrets the job had are left out, because a scheduled job can't take secrets. See On a schedule.

From the SDK

Workspace jobs need the privatemind SDK 0.6.0 or newer. The SDK is pre-installed and pre-configured in every workspace. There, import privatemind and call submit_job, and the home volume and the cluster default to the workspace's own. Outside a workspace, such as on a laptop or in CI, authenticate with an API key as the SDK overview describes and pass home_volume yourself. The job then runs on that volume's cluster unless you set target_cluster.

Submit a job

submit_job takes the command first and everything else as keyword arguments. It returns a Job handle.

Python
from privatemind import submit_job

job = submit_job(
    "python prepare.py --split val",
    template="python-slim-cpu",
    max_duration="2h",
)
job.wait()
print(job.phase, job.exit_code)   # Succeeded 0
print(job.logs())                 # the last 16 KiB of output
Parameter Meaning
command The shell command. Up to 4096 characters.
template The workspace template, by name.
template_version A version the template offers, such as "1.2.1". The current version when omitted.
home_volume The home volume, by name. Inside a workspace it defaults to the workspace's own; outside one it is required.
extra_volumes Up to 16 more volumes, by name.
weights Up to 8 versions from your org's registry, mounted read-only: ["credit-ft:2026-11-16"] appears at /home/privatemind/models/credit-ft, and [{"name": "base", "ref": "credit-ft:2026-11-16"}] at /home/privatemind/models/base. Pinned to the digest at submit. SDK 0.9.0 or later.
image A container your org pushed to its registry, such as "customers/your-org/trainer:v2", run in place of the template image. Needs gpus of 1 or more, cannot be combined with template_version, and is pinned to its digest at submit. The image runs as an unprivileged user and needs /usr/bin/tini, bash, mkdir and tee. SDK 0.9.0 or later.
target_cluster The GPU cluster to run on. Defaults to the workspace's cluster, then to the home volume's.
gpus Whole cards to hold, 0 to 8. Default 0.
gpu_placements Pin the cards: one {"host": ..., "indices": [...]} with exactly gpus indices, the shape submit_training takes. Omit it and the platform picks free cards on one machine.
cpu Cores: 2, 2.5 or "500m". The template's default when omitted.
memory A size in Mi, Gi or Ti, such as "16Gi". The template's default when omitted.
net_profile A network profile your org offers, by slug.
max_duration "24h", "90m", "2h30m", or a timedelta. 24 hours when omitted, at most seven days.
ttl_seconds_after_finished How long the record stays after the job ends, in seconds, up to seven days. One day when omitted.
shared True makes the job visible to the whole org.
name Your own name instead of a generated one: lowercase letters, digits and dashes, up to 40 characters, starting and ending with a letter or digit.
client An explicit Client to submit through.

Arguments are checked before any request is sent, so a bad value (a command over the limit, a seventeenth extra volume, a max_duration over seven days) raises ValidationError on your side. What the platform itself refuses is listed under Refused at submit.

The Job handle

The handle is a live view of the job. Its status properties read the last fetched record, and fetch one on first access if there is none.

Member What it gives you
job.wait(timeout=None, poll_interval=5.0, progress=False) Blocks until the job leaves Pending or Running, then returns the handle. Raises TimeoutError past timeout; the job itself keeps running. progress=True prints a line per phase change. on_change= and on_unknown_phase= work as they do on Run.wait.
job.refresh() Re-fetch the record. Returns the handle.
job.logs() The last 16 KiB of output, recorded when the job ended. Empty while it runs.
job.delete() Ends the job if it still runs and removes the record. It returns before the removal is complete, so a read straight after may still find the job (job.deleting is True then). A job already gone is not an error.
job.phase Pending, Running, Succeeded or Failed.
job.reason, job.message The platform's one-word reason for the phase, such as DeadlineExceeded, and its status text, when it has them.
job.exit_code The command's exit code once it exited. None before that, and on a job stopped at its deadline.
job.start_time, job.end_time, job.created_at RFC 3339 timestamps. created_at is the submit, which comes before start_time.
job.kind Always "job". A training run's is "training".
job.visibility "private", "shared" or "personal".
job.name, job.template, job.template_version, job.command, job.home_volume, job.extra_volumes, job.target_cluster, job.gpus, job.gpu_host, job.gpu_indices, job.cpu, job.memory, job.net_profile, job.max_duration, job.ttl_seconds_after_finished What the job was submitted with. cpu and memory are None when the template's defaults applied; net_profile is None on your org's default network.
Python
from privatemind import submit_job

job = submit_job(
    "python eval.py --ckpt checkpoints/last.pt",
    template="pytorch-gpu",
    gpus=1,
)
job.wait(timeout=3600, progress=True)
if job.phase == "Failed":
    print(job.reason, job.message)
    print(job.logs())

A Job holds an authenticated client, so it cannot be pickled or deep-copied. To act on a job from another process, pass its name and re-fetch the handle with get_workspace_job(name). In a notebook, a Job that is the last expression in a cell renders as a status table, from the record it last fetched.

List and fetch

list_jobs() returns every job you may see, of both kinds: a Run for a training run and a Job for a workspace job, each with a kind that says which. phase= filters on an exact, case-insensitive match. get_workspace_job(name) fetches one workspace job; get_job(name) is the training call, and list_training_jobs() lists training runs alone.

Python
from privatemind import list_jobs, get_workspace_job

for j in list_jobs(phase="Running"):
    print(j.kind, j.name, j.phase)

job = get_workspace_job("nightly-prep")
print(job.exit_code, job.logs())

Inside a job

A job gets its own PrivateMind key, injected the way a workspace's is, so from inside a job submit_training, submit_job and MLflow logging work with nothing to configure. A scheduled retrain can be a job whose command calls submit_training.

The key expires on its own after the job's maximum duration, rounded up to whole days, plus one day. It is also revoked when the platform next reads the finished job. A job you wait on with wait(), open in the Jobs tab, or run from an automation therefore loses its key as soon as it ends. A job nobody reads keeps it until it expires. Usage under that key counts against your budget.

Warning

A password or token typed into the command is stored in the job's record. If your organisation has Secrets on, attach the secret in the Run a job form and read it from /var/run/privatemind-secrets/NAME; submit_job doesn't take secrets. Otherwise, keep secrets in a file on the home volume with mode 0600 and read them from there.

On a schedule

In Run a job, pick On a schedule to run the job on a cron schedule, in a time zone you choose, at most every five minutes. The schedule keeps the job's settings, with the GPUs as a count: each run takes whichever cards are free when it starts. Each run is a new job in the Jobs tab, named sched-<schedule>-<run> and private to you. A scheduled job can't take secrets.

Your schedules are listed above the jobs, with their last run and their next one. From there you can pause, resume, edit or delete a schedule, and open its recent runs: the last 50, kept for 90 days, including runs that were skipped or could not start. Deleting a schedule doesn't stop a run that has already started. A run's output stays in ~/.privatemind/jobs/<name>/stdout.log on its home volume after the job's record has gone.

Each run is submitted as you, with the same checks as Run a job: your compute access, your quota, the volumes you may mount and, for a GPU job, a free card. A run that fails one of these doesn't start, and the runs list says why. If a run is still going when the next one is due, the next one is skipped. A resumed schedule waits for its next time and doesn't catch up on the runs it missed while paused.

A run that fails or can't start notifies you in the bell and by email, with at most one email an hour per schedule. When runs are skipped, the first skip in a row notifies you in the bell. Turn on Also notify me when a run succeeds to hear about every finished run.

An org admin can turn on View all jobs to see every schedule in the organisation, and can pause or delete any of them. Only its owner can change or resume a schedule, because its runs start as them.

In an automation

An automation can run a job as one of its steps. The automation itself can run on a schedule, on a webhook or on demand. The builder's Run Job step takes a job without a name: template and version, command, home and extra volumes, GPUs, CPU, memory and max duration. Tokens such as {{prev.output}} and {{vars.name}} in the command, or in a volume name, are filled in when the step runs, as in every other step.

Each run of the automation submits one job, named after the workflow, the run and the step, and waits for it to end. The wait survives a platform restart: a run that was in a job step is picked up where it stopped, and the job is never run twice. The step hands the job's record to the next step as its output:

JSON
{
  "name": "...",
  "namespace": "...",
  "phase": "Succeeded",
  "exitCode": 0,
  "logPath": "~/.privatemind/jobs/<name>/stdout.log",
  "logTail": "..."
}

namespace is where the platform ran the job; the other fields are the ones you will use.

A job that fails fails the step. The run history, the notification and the email show the job's status message (or its reason, when there is no message) and the last three lines of the output. Cancelling a run ends its job.

The job runs as the user the run bills: the owner for a scheduled firing, and the person who pressed Run now otherwise. That user's key is used, that user's quota is charged, and the volumes must be ones that user may mount. So a shared automation that names the owner's private home volume runs on its schedule, and refuses a colleague's Run now with the volume's name.

A job step suits work that runs and finishes. To keep an interactive workspace up at set hours, use the builder's Workspace step; see Stop and start without the browser.

Where next

  • Training jobs: training runs on the platform's training images, tracked in MLflow.
  • Workspaces: the image and the home volume a job runs on.
  • Secrets: passwords and API keys a job reads as files.
  • Cloud agents: automations, schedules and run history.
  • SDK overview: configuration, authentication and the exception hierarchy.