Get API key

Batch API

View as Markdown

Run bulk chat-completion work asynchronously at half the token cost. OpenAI-compatible.

The Batch API runs large sets of chat-completion requests asynchronously at 50% of the normal token rate. Use it for work that can wait: results can arrive at any point in the completion window.

It follows OpenAI's Batch API, so the official openai client works unchanged. Message Batches is the Anthropic-format version: requests go inline, with the same limits and pricing.

The flow

  1. Upload a JSONL file of requests with POST /v1/files (purpose: batch).
  2. Create a batch from that file with POST /v1/batches.
  3. Poll the batch with GET /v1/batches/{id} until it reaches a terminal status.
  4. Download the output file, and the error file if there is one, with GET /v1/files/{id}/content.

The input file

The input is a JSONL file with one request per line. Each line has a custom_id you choose, the method, the url and the body you would send to that endpoint.

JSON
{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "default", "messages": [{"role": "user", "content": "Summarise: ..."}]}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "default", "messages": [{"role": "user", "content": "Translate to French: ..."}]}}
  • url must match the batch's endpoint: /v1/chat/completions, or /v1/embeddings for an embeddings batch on an embeddings model.
  • custom_id must be unique within the file. Results are keyed by it, not by line order.
  • body is a normal chat-completion request. Set max_tokens per line if you need a non-default limit.

Upload the input file

curl -s "https://api.privatemind.com/v1/files" \
  -H "Authorization: Bearer $PMIND_KEY" \
  -F purpose=batch \
  -F file=@requests.jsonl
from openai import OpenAI

client = OpenAI(base_url="https://api.privatemind.com/v1", api_key=PMIND_KEY)
f = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
print(f.id)  # file-...

Keep the id of the returned file object for the next step.

Create a batch

curl -s "https://api.privatemind.com/v1/batches" \
  -H "Authorization: Bearer $PMIND_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file-abc123",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'
batch = client.batches.create(
    input_file_id=f.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)
print(batch.id, batch.status)  # batch_... in_progress
  • input_file_id (required): the id of an uploaded purpose: batch file you can see. The batch takes that file's visibility: if the file is private, only its owner sees the batch and its output and error files.
  • endpoint (required): /v1/chat/completions or /v1/embeddings. Requests must use a model of the matching type that GET /v1/models marks batch_eligible.
  • completion_window (required): must be 24h.
  • metadata (optional): string keys and values returned on the batch object. Up to 16 pairs, keys up to 64 characters, values up to 512.

The batch object

JSON
{
  "id": "batch_abc123",
  "object": "batch",
  "endpoint": "/v1/chat/completions",
  "input_file_id": "file-abc123",
  "completion_window": "24h",
  "status": "in_progress",
  "output_file_id": null,
  "error_file_id": null,
  "created_at": 1781481050,
  "expires_at": 1781567450,
  "request_counts": { "total": 2, "completed": 0, "failed": 0 },
  "errors": null,
  "metadata": null
}

status starts at in_progress, then moves to a terminal (final) status:

  • completed: the batch finished. output_file_id is set if any request succeeded, and error_file_id if any request errored.
  • failed: the input file was rejected as a whole, for example for invalid JSONL. See errors on the object.
  • expired: the completion window ended before every request ran. Completed requests are in the output file, the rest in the error file with the code batch_expired.
  • cancelling / cancelled: you cancelled the batch (see Cancel a batch).

request_counts tracks progress: total, completed (succeeded) and failed (errored). Cancelled and expired requests count in total only.

Check status

curl -s "https://api.privatemind.com/v1/batches/batch_abc123" \
  -H "Authorization: Bearer $PMIND_KEY"
import time

while batch.status not in ("completed", "failed", "expired", "cancelled"):
    time.sleep(30)
    batch = client.batches.retrieve(batch.id)
print(batch.status)

There is no push notification, so poll at a modest interval until status is terminal.

Download results

The output file is JSONL, with one line for each request that succeeded. Lines can come in any order, so match them on custom_id.

curl -s "https://api.privatemind.com/v1/files/file-out-batch_abc123/content" \
  -H "Authorization: Bearer $PMIND_KEY"
import json

if batch.output_file_id:
    content = client.files.content(batch.output_file_id).text
    for line in content.splitlines():
        row = json.loads(line)
        print(row["custom_id"], row["response"]["status_code"])

Each output line has a batch-scoped id, your custom_id, the HTTP response and an error that is null on success:

JSON
{"id": "batch_req_...", "custom_id": "req-1", "response": {"status_code": 200, "request_id": "req_...", "body": { "id": "chatcmpl-...", "choices": [ ... ] }}, "error": null}

Requests that errored are in the error file (error_file_id), in the same format with response null and error describing the failure.

Cancel a batch

cURL
curl -s -X POST "https://api.privatemind.com/v1/batches/batch_abc123/cancel" \
  -H "Authorization: Bearer $PMIND_KEY"

Requests already running finish, and the batch shows cancelling until they do. With none running, it goes straight to cancelled. Completed work is billed and is in the output file.

List batches

cURL
curl -s "https://api.privatemind.com/v1/batches?limit=20" \
  -H "Authorization: Bearer $PMIND_KEY"

GET /v1/batches returns one page of the batches you can see. Pass after=<last batch id> for the next page, until a page comes back empty. There is no status filter, so filter on status yourself. Message batches are not in this list.

When batch work runs

Batch work shares each model's capacity with live traffic, and PrivateMind keeps it from slowing live requests by more than a small margin. On a model nobody is using live, batch work takes all the capacity. While live requests run, it runs only as many requests as the model can take without a noticeable slowdown, possibly none. Requests with very long inputs wait until the model is quiet. If live load rises, some running batch requests go back in the queue, and you aren't charged for the stopped attempt. Your own live requests to the same model count as live traffic too.

Throughput follows live load, so it is highest overnight and at weekends. Each model has its own batch queue, so a backlog on one model does not hold up batches on another. On one model, your organisation's batches run oldest first, so splitting a job into more batches does not speed it up. Requests still queued when the window ends expire unbilled, ready to resubmit from the error file.

Limits and retention

  • Requests per batch: up to 10,000.
  • Rate of creation: each org can create 20 batches per hour and have 50,000 requests queued or running. Both caps are shared with Message Batches.
  • Input file size: up to 64 MB (the Files API ceiling).
  • Completion window: 24 hours. Requests not run by then expire.
  • Pricing: 50% of the model's normal per-token rate, for work that actually ran.
  • Retention: the batch's input, output and error files are kept for 29 days after the batch reaches a terminal status, then deleted. Download the results you want to keep before then.
Note

Batch input and output files hold your content on our servers, so the batch feature is not eligible for zero-data-retention (ZDR) arrangements.

Using the PrivateMind Python SDK

The PrivateMind Python SDK, pre-installed in every workspace, wraps this flow in helpers and a Batch object that waits and iterates over the results for you. See Batch in the Python SDK.

Python
from privatemind import Client

client = Client()  # base_url + token from the environment

f = client.create_file(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.create_batch(
    input_file_id=f.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)
batch.wait()                     # poll until terminal
for row in batch.results():      # one dict per output line
    print(row["custom_id"], row["response"]["status_code"])

Where next