The privatemind SDK runs bulk chat-completion work through the Batch API, billed at 50% of the normal token rate. The Batch API page has pricing and the server-side rules. The SDK gives you six functions and a Batch object that waits and iterates the results for you. The steps are the same as over HTTP:
- Upload a JSONL file of chat-completion requests.
- Create a batch from it.
- Wait for a terminal status.
- Read the output.
The examples run as they are in a PrivateMind workspace, where Client() reads its token and base URL from the environment. For configuration, authentication and errors, see the SDK overview.
The six calls
Each call maps to one HTTP endpoint. They are methods on Client and also module-level imports (from privatemind import create_batch). The module-level functions share one client per process, created on first use. Pass client= to any of them to use a Client of your own.
| Call | Endpoint |
|---|---|
create_file(file=, purpose="batch") |
POST /v1/files |
create_batch(input_file_id=, endpoint=, completion_window=, metadata=) |
POST /v1/batches |
get_batch(batch_id) |
GET /v1/batches/{id} |
list_batches(limit=, after=) |
GET /v1/batches |
cancel_batch(batch_id) |
POST /v1/batches/{id}/cancel |
get_file_content(file_id) |
GET /v1/files/{id}/content |
These arguments are checked in the SDK and raise ValidationError before any request is sent:
purposemust be"batch"(create_file)endpointmust be"/v1/chat/completions"andcompletion_windowmust be"24h"(create_batch)limitmust be between 1 and 100 (list_batches)
Build the input file
The input is JSONL: one JSON object per line, one line per request, in the shape the Batch API defines (custom_id, method, url, body). A loop of json.dumps calls builds it:
import json
docs = {
"doc-1": "First quarterly report text ...",
"doc-2": "Second quarterly report text ...",
"doc-3": "Third quarterly report text ...",
}
with open("requests.jsonl", "w") as fh:
for custom_id, text in docs.items():
fh.write(json.dumps({
"custom_id": custom_id,
"method": "POST",
"url": "/v1/chat/completions",
"body": {
"model": "default",
"messages": [{"role": "user", "content": f"Summarise in one paragraph:\n\n{text}"}],
},
}) + "\n")custom_id must be unique within the file, because results come back keyed by it. body is a normal chat-completion request; set max_tokens on a line if you need a limit other than the default.
Upload and create
from privatemind import Client
client = Client() # token and base URL from the workspace environment
f = client.create_file(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.create_batch(
input_file_id=f.id,
endpoint="/v1/chat/completions",
completion_window="24h",
)
print(batch.id, batch.status) # batch_... in_progresscreate_file uploads the file as multipart form data and returns a BatchFile (id, filename, bytes, purpose, status, created_at). create_batch takes that file's id and returns a Batch. The optional metadata= is a dict of string keys and values returned on the batch object. The Batch API page has its size limits.
Wait for the batch
batch.wait() # blocks until terminal, refreshes the object in placeBatch.wait() polls while status is one of the four live values: validating, in_progress, finalizing or cancelling. It stops on the four terminal ones: completed, failed, expired or cancelled. Each poll updates the object in place, and wait() returns the batch itself. To classify a status yourself, import the two sets as ACTIVE_BATCH_STATUSES and TERMINAL_BATCH_STATUSES.
poll_interval=is the starting number of seconds between polls (default 5). It doubles each poll, up to 30 seconds, or up to your starting interval if you set one higher than 30. Zero or a negative value raisesValueError.timeout=limits the wait, in seconds.None, the default, waits forever. When it runs out,wait()raises Python's built-inTimeoutErrorwith the last status it saw in the message.on_unknown_status=decides what happens if the server reports a status in neither set, which means your SDK is older than the platform. The default,"raise", raisesUnknownStatusError(carrying.batch_idand.status). It raises because the batch may still be running, and treating it as finished could letresults()return a partial file. Pass"return"andwait()returns with a warning.
A timeout stops only the waiting: the batch keeps running on the server. Wait again, or stop it with cancel_batch:
try:
batch.wait(timeout=3600)
except TimeoutError:
print("still running:", batch.status, batch.request_counts)Read the results
summaries = {}
if batch.output_file_id:
for row in batch.results():
body = row["response"]["body"]
summaries[row["custom_id"]] = body["choices"][0]["message"]["content"]
if batch.error_file_id:
for row in batch.error_results():
print("failed:", row["custom_id"], row["error"])results() downloads the output file and yields one plain dict per JSONL line (custom_id, response, error), as written in the file. error_results() does the same for the error file, which holds the requests that failed. Output order is not guaranteed, so collect by custom_id. Both download the whole file before reading it, and a malformed line raises PrivatemindError.
What each raises:
- Before the batch is terminal, both raise
PrivatemindErrortelling you towait()first. - A terminal batch with no output file means no request succeeded. If the whole input file was rejected,
results()anderror_results()both raise and point you tobatch.errors, which says why. Otherwiseresults()raises and points you toerror_results(). - A terminal batch with no error file had no request fail, and
error_results()yields nothing. On a cleanly completed batch, theif batch.error_file_id:check in the example above is optional.
To keep the raw file itself, download its bytes by id:
raw = client.get_file_content(batch.output_file_id)
with open("batch-output.jsonl", "wb") as fh:
fh.write(raw)The batch lifecycle
A batch starts in_progress and ends in one of the four terminal statuses, named as in the Batch API:
completed: finished.output_file_idis set, anderror_file_idtoo if any request errored.failed: the input file was rejected as a whole; the diagnosis is onbatch.errors.expired: the 24 hour window ran out first. Finished requests are in the output file, the rest in the error file.cancelled: you cancelled it. A batch with requests in flight finishes them first and showscancellingmeanwhile. Completed work is billed and stays readable in the output file.
Progress is in request_counts (total, completed, failed). Cancelled and expired requests count toward total only, never failed. Check it with get_batch:
b = client.get_batch(batch.id)
print(b.status, b.request_counts.completed, "of", b.request_counts.total)Pagination
list_batches returns one page as a plain list. It takes limit= (1 to 100) and after=, the id of the batch to start after. Pass after=page[-1].id for each next page until a page comes back empty:
batches = []
page = client.list_batches(limit=100)
while page:
batches.extend(page)
page = client.list_batches(limit=100, after=page[-1].id)The server can't filter by status, so filter in your code:
in_progress = [b for b in batches if b.status == "in_progress"]Called with neither limit= nor after=, list_batches() raises PrivatemindError if the server reports the list was truncated. The error tells you to pass after= for the next page. With either argument, the call returns the page without raising.
Handling errors
Every SDK exception subclasses PrivatemindError; the SDK overview has the hierarchy. The read calls (get_batch, list_batches, get_file_content and the polling inside wait()) retry temporary server errors on their own. The write calls (create_file, create_batch, cancel_batch) are never re-sent for you, so wrap them in your own retry where it matters:
import time
from privatemind import RateLimitError
for attempt in range(5):
try:
batch = client.create_batch(
input_file_id=f.id,
endpoint="/v1/chat/completions",
completion_window="24h",
)
break
except RateLimitError as exc:
time.sleep(exc.retry_after or 2 ** attempt)RateLimitError.retry_after holds the server's Retry-After header, or None if it sent none.
Limits and retention
The server enforces the request count, file size and rate limits, so they can change without an SDK upgrade. A rejected request raises a normal SDK exception. See limits and retention for the current numbers.
Batch input and output files hold your content and are kept on the server for up to 29 days. Batch is not eligible for zero-data-retention (ZDR) arrangements.
Where next
- Batch API: the JSONL line format, the batch object, limits, and pricing.
- Files: the endpoints behind
create_fileandget_file_content. - SDK overview: configuration, authentication, and the error hierarchy.