# How training works The open-source libraries a training job runs on, what PrivateMind adds on top, and the versions in the current training images. A training job on PrivateMind runs on standard open-source libraries, from NVIDIA's NCCL up to Hugging Face's TRL. PrivateMind adds the SDK that sets a job up and the platform that runs it on your org's GPUs. ## The stack, from the GPUs up - **NCCL** is NVIDIA's library for GPU-to-GPU communication. During training it moves gradients and weights between a job's GPUs. - **PyTorch**, in NVIDIA's CUDA build, runs the model's maths on the GPUs. It also starts one training process per GPU and uses NCCL to keep them in step. - **Hugging Face Transformers** loads the base model and its tokenizer, from a Hugging Face id, a folder on your volume or a registry version. TRL's trainers are built on its trainer. - **PEFT** adds a LoRA adapter: a small set of new weights trained on top of the frozen base model. For QLoRA, **bitsandbytes** first loads the base model in 4 bits, so a large model fits on fewer GPUs. - **TRL** holds the trainers for SFT, DPO, KTO and GRPO. A fine-tune's `task` picks one. - **Accelerate** lets the trainer run across all of a job's GPUs. For a full fine-tune, **DeepSpeed ZeRO-3** splits the model, its gradients and its optimizer state across those GPUs, so a model too large for one GPU can still train. - **vLLM** generates text from a model. GRPO uses it to write the answers it scores at each step, `pm.evaluate` uses it to answer your eval set, and model servers use it to serve your models and adapters. - **Ray** gives each job its own small cluster: a head with no GPU that runs your entrypoint, and GPU workers that train. **KubeRay** runs that cluster on Kubernetes, and the **KAI** scheduler starts the head and every worker at once, so a job never holds some of its GPUs while it waits for the rest. - **MLflow** records each run: its configuration, the trainer's metrics as it trains and, for an adapter, the finished files. Each org that uses MLflow has its own instance. ## What PrivateMind adds - **The `privatemind` SDK.** `pm.finetune` checks your configuration and dataset, then submits the job. `pm.evaluate` scores a model on your eval set. `run.publish` merges an adapter into its base model and pushes the result to your org's registry. A [model server](https://docs.privatemind.com/sdk-model-servers.html) serves a published version, or a base model with your LoRA adapters on it. - **The recipes.** Three Python scripts drive the libraries above: one fine-tunes with TRL, one evaluates with vLLM and one merges an adapter for publishing. They come inside the SDK package, so you can read what your job runs. - **The platform.** It turns a job into a Ray cluster on your org's GPUs, in your org's own namespace, and tracks it until it ends. ## How a job runs 1. You submit the job from the SDK, or with **New run** on the Training tab of the app. A fine-tune first checks its configuration and dataset where you call it, then copies the recipe and its configuration to your home volume. 2. The platform checks the request against your org's GPUs, your [personal compute quota](https://docs.privatemind.com/training.html#personal-compute-quotas) and the list of training images, and records the job. 3. It starts the job's Ray cluster in your org's namespace, on free GPUs your org owns or the ones you named. While the GPUs are busy, the run shows `Queued`. 4. The Ray head runs the recipe. The recipe starts one training process per GPU, and each loads the model with Transformers and trains it with TRL. 5. The trainer writes checkpoints and the result to `output_path` on your volume, and sends its metrics to your org's MLflow. The run ends as `Succeeded`, `Failed` or `Stopped`, which the [`Run` handle](https://docs.privatemind.com/training-sdk.html#the-run-handle) reports. 6. From there, `run.publish` puts the result in your org's registry, and a model server can serve it. ## Versions in the current images A fine-tune runs on one of two training images. SFT, DPO and KTO use the default image, `workspace-pytorch-ngc`, the same image a PyTorch GPU workspace runs. GRPO and `pm.evaluate` use `training-rl`, which adds vLLM. | Library | `workspace-pytorch-ngc` 1.4.40 | `training-rl` 0.1.0 | |---|---|---| | Base image | NVIDIA PyTorch container 26.10 | NVIDIA vLLM container 26.10 | | CUDA | 13.4 | 13.4 | | NCCL | 2.32.3 | 2.32.3 | | PyTorch | 2.15.0a0+875d815, NVIDIA's build | 2.15.0a0+875d815, NVIDIA's build | | Transformers | 5.23.0 | 5.17.0 | | PEFT | 0.21.3 | 0.21.3 | | bitsandbytes | 0.50.2 | 0.50.2 | | TRL | 1.18.0 | 1.18.1 | | Accelerate | 1.15.0 | 1.15.0 | | DeepSpeed | 0.20.0 | 0.20.0 | | vLLM | not included | 0.31.0 | | Ray | 2.59.0 | 2.59.0 | | MLflow | 3.18.0 | 3.18.0 | | `privatemind` SDK | 0.11.0 | 0.11.0 | Leave `image` unset on `pm.finetune` or `pm.evaluate` and the job runs on the newest version of the image it needs. To pin a version, `Client().list_training_images()` lists every version on offer for both images, the default first: pass an entry's `value` as `image=`. In the app, the **Image** field of a new training run offers the same list. [Submit a job](https://docs.privatemind.com/training-sdk.html#submit-a-job) has the details. Before training starts, a fine-tune's log names the version of each library in the image. Model servers run vLLM in a version you pick from `pm.model_server_catalog().versions`. ## Your data and weights What a job reads and writes stays in your org: - Your dataset is read from your volume. A reward function or grader you write is saved to your home volume and runs inside your own job, on your GPUs. - Checkpoints and the finished adapter or model are written to `output_path` on your volume. - `run.publish` pushes to your org's [registry](https://docs.privatemind.com/registry.html), which only your org can reach. - Parameters, metrics and adapter files go to your org's MLflow. The trainer reports to no other service. A base model given as a Hugging Face id is downloaded from Hugging Face when the job starts. To start from weights you already hold, pass a path on your volume as `model=`, or a registry version as `weights=`. ## Open source, no lock-in Each library in the stack above is open source, and the SDK, recipes included, is Apache 2.0 licensed. An adapter is a standard PEFT folder: `adapter_config.json`, `adapter_model.safetensors` and the tokenizer files. Any tool that reads PEFT adapters can load it, on PrivateMind or elsewhere. A published model is a standard Hugging Face model folder with safetensors weights, and `privatemind weights pull` downloads any version to a folder you choose. To train with other libraries or other versions, run your own code as a [workspace job](https://docs.privatemind.com/sdk-jobs.html), in a workspace image or a container your org pushed to its registry. ## Where next - [Compute and training overview](https://docs.privatemind.com/training.html): what a GPU allocation gives you, and personal compute quotas. - [Fine-tuning](https://docs.privatemind.com/sdk-finetune.html): SFT, methods, checkpoints and resources. - [Preference tuning](https://docs.privatemind.com/sdk-preference-tuning.html): DPO and KTO on pairs of answers. - [Reinforcement learning](https://docs.privatemind.com/sdk-grpo.html): GRPO with a reward function you write in Python. - [Evaluate and publish](https://docs.privatemind.com/sdk-evaluate.html): score a model, push it to your registry and serve it. - [Workspace jobs](https://docs.privatemind.com/sdk-jobs.html): one command in a workspace image or your own container. - [Model servers](https://docs.privatemind.com/sdk-model-servers.html): serve a published model, or a base model with LoRA adapters.