# Batch API Run bulk chat-completion work asynchronously at half the token cost. OpenAI-compatible. The Batch API runs large sets of chat-completion requests asynchronously at **50% of the normal token rate**. Use it for work that can wait: results can arrive at any point in the completion window. It follows OpenAI's Batch API, so the official `openai` client works unchanged. [Message Batches](https://docs.privatemind.com/message-batches.html) is the Anthropic-format version: requests go inline, with the same limits and pricing. ## The flow 1. Upload a JSONL file of requests with `POST /v1/files` (`purpose: batch`). 2. Create a batch from that file with `POST /v1/batches`. 3. Poll the batch with `GET /v1/batches/{id}` until it reaches a terminal status. 4. Download the output file, and the error file if there is one, with `GET /v1/files/{id}/content`. ## The input file The input is a [JSONL](https://jsonlines.org/) file with one request per line. Each line has a `custom_id` you choose, the `method`, the `url` and the `body` you would send to that endpoint. ```json {"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "default", "messages": [{"role": "user", "content": "Summarise: ..."}]}} {"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "default", "messages": [{"role": "user", "content": "Translate to French: ..."}]}} ``` - `url` must match the batch's `endpoint`: `/v1/chat/completions`, or `/v1/embeddings` for an embeddings batch on an embeddings model. - `custom_id` must be unique within the file. Results are keyed by it, not by line order. - `body` is a normal chat-completion request. Set `max_tokens` per line if you need a non-default limit. ## Upload the input file **cURL** ```bash curl -s "https://api.privatemind.com/v1/files" \ -H "Authorization: Bearer $PMIND_KEY" \ -F purpose=batch \ -F file=@requests.jsonl ``` **Python** ```python from openai import OpenAI client = OpenAI(base_url="https://api.privatemind.com/v1", api_key=PMIND_KEY) f = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch") print(f.id) # file-... ``` Keep the `id` of the returned file object for the next step. ## Create a batch **cURL** ```bash curl -s "https://api.privatemind.com/v1/batches" \ -H "Authorization: Bearer $PMIND_KEY" \ -H "Content-Type: application/json" \ -d '{ "input_file_id": "file-abc123", "endpoint": "/v1/chat/completions", "completion_window": "24h" }' ``` **Python** ```python batch = client.batches.create( input_file_id=f.id, endpoint="/v1/chat/completions", completion_window="24h", ) print(batch.id, batch.status) # batch_... in_progress ``` - **`input_file_id`** (required): the `id` of an uploaded `purpose: batch` file you can see. The batch takes that file's [visibility](https://docs.privatemind.com/files.html#who-can-see-a-file): if the file is private, only its owner sees the batch and its output and error files. - **`endpoint`** (required): `/v1/chat/completions` or `/v1/embeddings`. Requests must use a model of the matching type that `GET /v1/models` marks `batch_eligible`. - **`completion_window`** (required): must be `24h`. - **`metadata`** (optional): string keys and values returned on the batch object. Up to 16 pairs, keys up to 64 characters, values up to 512. ## The batch object ```json { "id": "batch_abc123", "object": "batch", "endpoint": "/v1/chat/completions", "input_file_id": "file-abc123", "completion_window": "24h", "status": "in_progress", "output_file_id": null, "error_file_id": null, "created_at": 1781481050, "expires_at": 1781567450, "request_counts": { "total": 2, "completed": 0, "failed": 0 }, "errors": null, "metadata": null } ``` `status` starts at `in_progress`, then moves to a terminal (final) status: - **`completed`**: the batch finished. `output_file_id` is set if any request succeeded, and `error_file_id` if any request errored. - **`failed`**: the input file was rejected as a whole, for example for invalid JSONL. See `errors` on the object. - **`expired`**: the completion window ended before every request ran. Completed requests are in the output file, the rest in the error file with the code `batch_expired`. - **`cancelling`** / **`cancelled`**: you cancelled the batch (see [Cancel a batch](https://docs.privatemind.com/batch.html#cancel-a-batch)). `request_counts` tracks progress: `total`, `completed` (succeeded) and `failed` (errored). Cancelled and expired requests count in `total` only. ## Check status **cURL** ```bash curl -s "https://api.privatemind.com/v1/batches/batch_abc123" \ -H "Authorization: Bearer $PMIND_KEY" ``` **Python** ```python import time while batch.status not in ("completed", "failed", "expired", "cancelled"): time.sleep(30) batch = client.batches.retrieve(batch.id) print(batch.status) ``` There is no push notification, so poll at a modest interval until `status` is terminal. ## Download results The output file is JSONL, with one line for each request that succeeded. Lines can come in any order, so match them on `custom_id`. **cURL** ```bash curl -s "https://api.privatemind.com/v1/files/file-out-batch_abc123/content" \ -H "Authorization: Bearer $PMIND_KEY" ``` **Python** ```python import json if batch.output_file_id: content = client.files.content(batch.output_file_id).text for line in content.splitlines(): row = json.loads(line) print(row["custom_id"], row["response"]["status_code"]) ``` Each output line has a batch-scoped `id`, your `custom_id`, the HTTP `response` and an `error` that is null on success: ```json {"id": "batch_req_...", "custom_id": "req-1", "response": {"status_code": 200, "request_id": "req_...", "body": { "id": "chatcmpl-...", "choices": [ ... ] }}, "error": null} ``` Requests that errored are in the **error file** (`error_file_id`), in the same format with `response` null and `error` describing the failure. ## Cancel a batch ```bash curl -s -X POST "https://api.privatemind.com/v1/batches/batch_abc123/cancel" \ -H "Authorization: Bearer $PMIND_KEY" ``` Requests already running finish, and the batch shows `cancelling` until they do. With none running, it goes straight to `cancelled`. Completed work is billed and is in the output file. ## List batches ```bash curl -s "https://api.privatemind.com/v1/batches?limit=20" \ -H "Authorization: Bearer $PMIND_KEY" ``` `GET /v1/batches` returns one page of the batches you can see. Pass `after=` for the next page, until a page comes back empty. There is no status filter, so filter on `status` yourself. [Message batches](https://docs.privatemind.com/message-batches.html) are not in this list. ## When batch work runs Batch work shares each model's capacity with live traffic, and PrivateMind keeps it from slowing live requests by more than a small margin. On a model nobody is using live, batch work takes all the capacity. While live requests run, it runs only as many requests as the model can take without a noticeable slowdown, possibly none. Requests with very long inputs wait until the model is quiet. If live load rises, some running batch requests go back in the queue, and you aren't charged for the stopped attempt. Your own live requests to the same model count as live traffic too. Throughput follows live load, so it is highest overnight and at weekends. Each model has its own batch queue, so a backlog on one model does not hold up batches on another. On one model, your organisation's batches run oldest first, so splitting a job into more batches does not speed it up. Requests still queued when the window ends expire unbilled, ready to resubmit from the error file. ## Limits and retention - **Requests per batch**: up to 10,000. - **Rate of creation**: each org can create 20 batches per hour and have 50,000 requests queued or running. Both caps are shared with [Message Batches](https://docs.privatemind.com/message-batches.html). - **Input file size**: up to 64 MB (the [Files API](https://docs.privatemind.com/files.html) ceiling). - **Completion window**: 24 hours. Requests not run by then expire. - **Pricing**: 50% of the model's normal per-token rate, for work that actually ran. - **Retention**: the batch's input, output and error files are kept for **29 days** after the batch reaches a terminal status, then deleted. Download the results you want to keep before then. > [!NOTE] > Batch input and output files hold your content on our servers, so the batch feature is not eligible for zero-data-retention (ZDR) arrangements. ## Using the PrivateMind Python SDK The PrivateMind Python SDK, pre-installed in every workspace, wraps this flow in helpers and a `Batch` object that waits and iterates over the results for you. See [Batch in the Python SDK](https://docs.privatemind.com/sdk-batch.html). ```python from privatemind import Client client = Client() # base_url + token from the environment f = client.create_file(file=open("requests.jsonl", "rb"), purpose="batch") batch = client.create_batch( input_file_id=f.id, endpoint="/v1/chat/completions", completion_window="24h", ) batch.wait() # poll until terminal for row in batch.results(): # one dict per output line print(row["custom_id"], row["response"]["status_code"]) ``` ## Where next - [Chat completions](https://docs.privatemind.com/chat-completions.html) for the request body each batch line carries. - [Message Batches](https://docs.privatemind.com/message-batches.html) for the Anthropic-shaped equivalent. - [Files](https://docs.privatemind.com/files.html) for uploading input files and downloading results by id. - [Budgets and limits](https://docs.privatemind.com/budgets-and-limits.html) for how the 50% batch rate counts against your budget. - [Errors](https://docs.privatemind.com/errors.html) for status codes and error shapes.