Workspace jobs

What a workspace job is

A workspace is an environment you sit in: a template image, a home volume mounted as your home directory, and a notebook, an editor and a terminal on top. A workspace job is that same environment with nothing on top. You give it one shell command. The platform starts the template image, mounts your home volume, runs the command as you in your home directory, and stops when the command does. The command's exit code becomes the job's exit code, and the output is saved on the home volume.

Because it is the same image and the same volume, your files are where a workspace would put them. A script at ~/projects/etl/run.py in a workspace is at ~/projects/etl/run.py in a job, and whatever the job writes is there the next time you open a workspace on that volume.

Two things are different from typing the command into a workspace terminal:

• There is no login shell. The command runs with the container's environment, and nothing in .bashrc, .zshrc or .profile is read. If the command needs a directory, a variable or an activated environment, put the cd, the export or the source in the command itself.
• There is a deadline. Every job has a maximum duration: 24 hours unless you set one, and at most seven days. When it passes, the platform stops the job.

The command can be up to 4096 characters. Anything longer belongs in a script on the home volume, with the command calling the script.

When to use one

Reach for a workspace job when you have something that runs on its own and you do not want to keep a workspace open for it: a data preparation step, an evaluation over a checkpoint, a script that rebuilds a dataset overnight. If it runs as a script from the workspace terminal, it runs as a job. A job can run on the home volume of a workspace that is open at the same time, and both see the same files live.

A training job is the other way to run work without a notebook, and it answers a different need. A training job runs across several GPU workers, on the platform's training images, with every run tracked in MLflow. A workspace job is one process in one place, in the environment you already work in, with an exit code and a log. When the work is one command in your own environment, a workspace job is the lighter tool. When it needs several GPU workers or experiment tracking, use a training job. A workspace job can also submit a training job, so a scheduled retrain can be a small job whose command calls submit_training (see Inside a job).

What happens when it runs

A job moves through four phases: Pending, Running, then Succeeded or Failed.

While a job is Pending, the platform is finding it a place to run and pulling the template image. The maximum duration is already counting: it starts when the job starts trying to run, not when the command begins. In Running, the command is executing. Succeeded means the command exited with code 0. Failed means anything else: a non-zero exit code, a job the platform stopped, or one that could not start. A failed job carries a reason and a message that say which. The exit code is present only when the command actually exited, so a job stopped at its deadline has none.

A job that passes its maximum duration is stopped and ends Failed with the reason DeadlineExceeded. A job whose pod no machine could take for a minute ends Failed with the reason Unschedulable and the scheduler's message, rather than waiting for its deadline.

Any GPUs the job asked for are held for its whole life, from submit until it ends, whether or not the command is using them.

After a job ends, its record (phase, exit code, the tail of the output and the settings it ran with) stays for one day and is then removed. You can ask for a longer life at submit, up to seven days. The output file on the home volume stays as long as the volume does.

Seeing what happened

The job's output is written to the home volume at ~/.privatemind/jobs/<name>/stdout.log as it runs. Open a workspace on the same volume and you can read the file, or follow it with tail -f, while the job is still going.

When the job ends, the last 16 KiB of that output are recorded on the job itself. That tail is what the Jobs tab shows when you open a job, and what logs() returns in the SDK. It is recorded at the end, so while a job is running the tail is empty and the file is the place to look. A job stopped at its deadline usually has no tail either, because the platform could not read it in time; the file on the home volume is the record then.

Volumes and who can see a job

A job mounts your home volume, and up to 16 more volumes by name, the way a workspace mounts them. The home volume must already exist; a job does not create one.

A job is private to you unless you submit it as shared, which makes it visible to the whole org, as a shared workspace is. A job that mounts a personal volume stays personal whatever you ask. A volume that belongs to another member and is not shared with you cannot be mounted at all: the submit is refused with the volume's name.

GPUs

Ask for 0 to 8 whole cards; they are held for the whole job. In the Jobs tab you choose a host and tick the cards on it, from the ones your org owns and nothing is using. From the SDK or an automation you can name the cards the same way with gpu_placements, or give a count and let the platform pick free cards your org owns on one machine. There is no queue: if no card is free the submit is refused right away, and a card something else holds is refused with the name of the workload holding it.

Refused at submit

The platform checks a job when you submit it and refuses with a message that says what is wrong:

• the home volume does not exist;
• a volume belongs to another member and is not shared with you (volume '<name>' is private to another member);
• the template, or the version you asked for, is not one your org offers;
• the name is already in use;
• no card is free, or a card you pinned is busy (the message names the workload holding it);
• your personal quota would be exceeded;
• a shared job names a network profile that is restricted to some members.

From the Jobs tab

The Jobs tab, under Inference & Training in your org's settings, lists the workspace jobs and training runs you may see in one table, each with its kind, phase, exit code and duration. Open a job for its output.

Run a job asks for the same things submit_job takes: a template and version, the command, the home volume and any extra volumes, the cards, CPU and memory, a maximum duration, and whether the job is shared. The form explains each field, and the table under Submit a job says what each one means. Delete ends a running job and removes its record. Schedule this on a job row opens the automation builder with that job filled in as a step, so a job you ran once by hand can become a schedule without retyping it. See On a schedule.

From the SDK

Workspace jobs are part of the privatemind SDK, 0.6.0 or newer. The SDK is pre-installed and pre-configured in every GPU workspace; there, import privatemind and call submit_job, and the home volume and the cluster default to the workspace's own. Outside a workspace (a laptop, CI), authenticate with an API key as the SDK overview describes and pass home_volume yourself; the job runs on that volume's cluster unless you set target_cluster.

Submit a job

submit_job takes the command first and everything else as keyword arguments. It returns a Job handle.

from privatemind import submit_job

job = submit_job(
    "python prepare.py --split val",
    template="python-slim-cpu",
    max_duration="2h",
)
job.wait()
print(job.phase, job.exit_code)   # Succeeded 0
print(job.logs())                 # the last 16 KiB of output

Arguments are checked before any request is sent, so a bad value (a command over the limit, a seventeenth extra volume, a max_duration over seven days) raises ValidationError on your side. What the platform itself refuses is listed under Refused at submit.

The Job handle

The handle is a live view of the job. Its status properties read the last fetched record, and fetch one on first access if there is none.

from privatemind import submit_job

job = submit_job(
    "python eval.py --ckpt checkpoints/last.pt",
    template="pytorch-gpu",
    gpus=1,
)
job.wait(timeout=3600, progress=True)
if job.phase == "Failed":
    print(job.reason, job.message)
    print(job.logs())

A Job holds an authenticated client, so it cannot be pickled or deep-copied. To act on a job from another process, pass its name and re-fetch the handle with get_workspace_job(name). In a notebook, a Job that is the last expression in a cell renders as a status table, from the record it last fetched.

List and fetch

list_jobs() returns every job you may see, of both kinds: a Run for a training run and a Job for a workspace job, each with a kind that says which. phase= filters on an exact, case-insensitive match. get_workspace_job(name) fetches one workspace job; get_job(name) is the training call, and list_training_jobs() lists training runs alone.

from privatemind import list_jobs, get_workspace_job

for j in list_jobs(phase="Running"):
    print(j.kind, j.name, j.phase)

job = get_workspace_job("nightly-prep")
print(job.exit_code, job.logs())

Inside a job

A job gets its own PrivateMind key, injected the way a workspace's is, so from inside a job submit_training, submit_job and MLflow logging work with nothing to configure. A scheduled retrain can be a job whose command calls submit_training.

The key expires on its own: the job's maximum duration rounded up to whole days, plus one day. It is also revoked when the platform next reads the finished job, so a job you wait on with wait(), open in the Jobs tab, or run from an automation loses its key as soon as it ends, and a job nobody reads keeps it until it expires. Usage under that key counts against your budget.

On a schedule

An automation can run a job on a schedule, on a webhook, or on demand. The builder has a Run Job step whose configuration is a job without a name: template and version, command, home and extra volumes, GPUs, CPU, memory and max duration. Tokens such as {{prev.output}} and {{vars.name}} in the command, or in a volume name, are filled in when the step runs, as in every other step. Schedule this on a job in the Jobs tab opens the builder with that job already in place.

Each run of the automation submits one job, named after the workflow, the run and the step, and waits for it to end. The wait survives a platform restart: a run that was in a job step is picked up where it stopped, and the job is never run twice. The step hands the job's record to the next step as its output:

{
  "name": "...",
  "namespace": "...",
  "phase": "Succeeded",
  "exitCode": 0,
  "logPath": "~/.privatemind/jobs/<name>/stdout.log",
  "logTail": "..."
}

namespace is where the platform ran the job; the other fields are the ones you will use.

A job that fails fails the step. The run history, the notification and the email show the job's status message (or its reason, when there is no message) and the last three lines of the output. Cancelling a run ends its job.

The job runs as the user the run bills: the owner for a scheduled firing, and the person who pressed Run now otherwise. That user's key is used, that user's quota is charged, and the volumes must be ones that user may mount. So a shared automation that names the owner's private home volume runs on its schedule, and refuses a colleague's Run now with the volume's name.

Where next

• Training jobs: distributed training across workers, tracked in MLflow.
• Workspaces: the image and the home volume a job runs on.
• Cloud agents: automations, schedules and run history.
• SDK overview: configuration, authentication and the exception hierarchy.