The Batch API runs large sets of chat-completion requests asynchronously at 50% of the normal token rate. Use it for work that can wait: results can arrive at any point in the completion window.
It follows OpenAI's Batch API, so the official openai client works unchanged. Message Batches is the Anthropic-format version: requests go inline, with the same limits and pricing.
The flow
- Upload a JSONL file of requests with
POST /v1/files(purpose: batch). - Create a batch from that file with
POST /v1/batches. - Poll the batch with
GET /v1/batches/{id}until it reaches a terminal status. - Download the output file, and the error file if there is one, with
GET /v1/files/{id}/content.
The input file
The input is a JSONL file with one request per line. Each line has a custom_id you choose, the method, the url and the body you would send to that endpoint.
{"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "default", "messages": [{"role": "user", "content": "Summarise: ..."}]}}
{"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "default", "messages": [{"role": "user", "content": "Translate to French: ..."}]}}urlmust match the batch'sendpoint:/v1/chat/completions, or/v1/embeddingsfor an embeddings batch on an embeddings model.custom_idmust be unique within the file. Results are keyed by it, not by line order.bodyis a normal chat-completion request. Setmax_tokensper line if you need a non-default limit.
Upload the input file
curl -s "https://api.privatemind.com/v1/files" \
-H "Authorization: Bearer $PMIND_KEY" \
-F purpose=batch \
-F file=@requests.jsonlfrom openai import OpenAI
client = OpenAI(base_url="https://api.privatemind.com/v1", api_key=PMIND_KEY)
f = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
print(f.id) # file-...Keep the id of the returned file object for the next step.
Create a batch
curl -s "https://api.privatemind.com/v1/batches" \
-H "Authorization: Bearer $PMIND_KEY" \
-H "Content-Type: application/json" \
-d '{
"input_file_id": "file-abc123",
"endpoint": "/v1/chat/completions",
"completion_window": "24h"
}'batch = client.batches.create(
input_file_id=f.id,
endpoint="/v1/chat/completions",
completion_window="24h",
)
print(batch.id, batch.status) # batch_... in_progressinput_file_id(required): theidof an uploadedpurpose: batchfile you can see. The batch takes that file's visibility: if the file is private, only its owner sees the batch and its output and error files.endpoint(required):/v1/chat/completionsor/v1/embeddings. Requests must use a model of the matching type thatGET /v1/modelsmarksbatch_eligible.completion_window(required): must be24h.metadata(optional): string keys and values returned on the batch object. Up to 16 pairs, keys up to 64 characters, values up to 512.
The batch object
{
"id": "batch_abc123",
"object": "batch",
"endpoint": "/v1/chat/completions",
"input_file_id": "file-abc123",
"completion_window": "24h",
"status": "in_progress",
"output_file_id": null,
"error_file_id": null,
"created_at": 1781481050,
"expires_at": 1781567450,
"request_counts": { "total": 2, "completed": 0, "failed": 0 },
"errors": null,
"metadata": null
}status starts at in_progress, then moves to a terminal (final) status:
completed: the batch finished.output_file_idis set if any request succeeded, anderror_file_idif any request errored.failed: the input file was rejected as a whole, for example for invalid JSONL. Seeerrorson the object.expired: the completion window ended before every request ran. Completed requests are in the output file, the rest in the error file with the codebatch_expired.cancelling/cancelled: you cancelled the batch (see Cancel a batch).
request_counts tracks progress: total, completed (succeeded) and failed (errored). Cancelled and expired requests count in total only.
Check status
curl -s "https://api.privatemind.com/v1/batches/batch_abc123" \
-H "Authorization: Bearer $PMIND_KEY"import time
while batch.status not in ("completed", "failed", "expired", "cancelled"):
time.sleep(30)
batch = client.batches.retrieve(batch.id)
print(batch.status)There is no push notification, so poll at a modest interval until status is terminal.
Download results
The output file is JSONL, with one line for each request that succeeded. Lines can come in any order, so match them on custom_id.
curl -s "https://api.privatemind.com/v1/files/file-out-batch_abc123/content" \
-H "Authorization: Bearer $PMIND_KEY"import json
if batch.output_file_id:
content = client.files.content(batch.output_file_id).text
for line in content.splitlines():
row = json.loads(line)
print(row["custom_id"], row["response"]["status_code"])Each output line has a batch-scoped id, your custom_id, the HTTP response and an error that is null on success:
{"id": "batch_req_...", "custom_id": "req-1", "response": {"status_code": 200, "request_id": "req_...", "body": { "id": "chatcmpl-...", "choices": [ ... ] }}, "error": null}Requests that errored are in the error file (error_file_id), in the same format with response null and error describing the failure.
Cancel a batch
curl -s -X POST "https://api.privatemind.com/v1/batches/batch_abc123/cancel" \
-H "Authorization: Bearer $PMIND_KEY"Requests already running finish, and the batch shows cancelling until they do. With none running, it goes straight to cancelled. Completed work is billed and is in the output file.
List batches
curl -s "https://api.privatemind.com/v1/batches?limit=20" \
-H "Authorization: Bearer $PMIND_KEY"GET /v1/batches returns one page of the batches you can see. Pass after=<last batch id> for the next page, until a page comes back empty. There is no status filter, so filter on status yourself. Message batches are not in this list.
When batch work runs
Batch work shares each model's capacity with live traffic, and PrivateMind keeps it from slowing live requests by more than a small margin. On a model nobody is using live, batch work takes all the capacity. While live requests run, it runs only as many requests as the model can take without a noticeable slowdown, possibly none. Requests with very long inputs wait until the model is quiet. If live load rises, some running batch requests go back in the queue, and you aren't charged for the stopped attempt. Your own live requests to the same model count as live traffic too.
Throughput follows live load, so it is highest overnight and at weekends. Each model has its own batch queue, so a backlog on one model does not hold up batches on another. On one model, your organisation's batches run oldest first, so splitting a job into more batches does not speed it up. Requests still queued when the window ends expire unbilled, ready to resubmit from the error file.
Limits and retention
- Requests per batch: up to 10,000.
- Rate of creation: each org can create 20 batches per hour and have 50,000 requests queued or running. Both caps are shared with Message Batches.
- Input file size: up to 64 MB (the Files API ceiling).
- Completion window: 24 hours. Requests not run by then expire.
- Pricing: 50% of the model's normal per-token rate, for work that actually ran.
- Retention: the batch's input, output and error files are kept for 29 days after the batch reaches a terminal status, then deleted. Download the results you want to keep before then.
Batch input and output files hold your content on our servers, so the batch feature is not eligible for zero-data-retention (ZDR) arrangements.
Using the PrivateMind Python SDK
The PrivateMind Python SDK, pre-installed in every workspace, wraps this flow in helpers and a Batch object that waits and iterates over the results for you. See Batch in the Python SDK.
from privatemind import Client
client = Client() # base_url + token from the environment
f = client.create_file(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.create_batch(
input_file_id=f.id,
endpoint="/v1/chat/completions",
completion_window="24h",
)
batch.wait() # poll until terminal
for row in batch.results(): # one dict per output line
print(row["custom_id"], row["response"]["status_code"])Where next
- Chat completions for the request body each batch line carries.
- Message Batches for the Anthropic-shaped equivalent.
- Files for uploading input files and downloading results by id.
- Budgets and limits for how the 50% batch rate counts against your budget.
- Errors for status codes and error shapes.