Get API key

Batch

View as Markdown

Run bulk chat-completion work from Python: build the JSONL input, upload it, create a batch, wait, and iterate the results at half the token rate.

The privatemind SDK runs bulk chat-completion work through the Batch API, billed at 50% of the normal token rate. The Batch API page has pricing and the server-side rules. The SDK gives you six functions and a Batch object that waits and iterates the results for you. The steps are the same as over HTTP:

  1. Upload a JSONL file of chat-completion requests.
  2. Create a batch from it.
  3. Wait for a terminal status.
  4. Read the output.

The examples run as they are in a PrivateMind workspace, where Client() reads its token and base URL from the environment. For configuration, authentication and errors, see the SDK overview.

The six calls

Each call maps to one HTTP endpoint. They are methods on Client and also module-level imports (from privatemind import create_batch). The module-level functions share one client per process, created on first use. Pass client= to any of them to use a Client of your own.

Call Endpoint
create_file(file=, purpose="batch") POST /v1/files
create_batch(input_file_id=, endpoint=, completion_window=, metadata=) POST /v1/batches
get_batch(batch_id) GET /v1/batches/{id}
list_batches(limit=, after=) GET /v1/batches
cancel_batch(batch_id) POST /v1/batches/{id}/cancel
get_file_content(file_id) GET /v1/files/{id}/content

These arguments are checked in the SDK and raise ValidationError before any request is sent:

  • purpose must be "batch" (create_file)
  • endpoint must be "/v1/chat/completions" and completion_window must be "24h" (create_batch)
  • limit must be between 1 and 100 (list_batches)

Build the input file

The input is JSONL: one JSON object per line, one line per request, in the shape the Batch API defines (custom_id, method, url, body). A loop of json.dumps calls builds it:

Python
import json

docs = {
    "doc-1": "First quarterly report text ...",
    "doc-2": "Second quarterly report text ...",
    "doc-3": "Third quarterly report text ...",
}

with open("requests.jsonl", "w") as fh:
    for custom_id, text in docs.items():
        fh.write(json.dumps({
            "custom_id": custom_id,
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {
                "model": "default",
                "messages": [{"role": "user", "content": f"Summarise in one paragraph:\n\n{text}"}],
            },
        }) + "\n")

custom_id must be unique within the file, because results come back keyed by it. body is a normal chat-completion request; set max_tokens on a line if you need a limit other than the default.

Upload and create

Python
from privatemind import Client

client = Client()   # token and base URL from the workspace environment

f = client.create_file(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.create_batch(
    input_file_id=f.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)
print(batch.id, batch.status)   # batch_... in_progress

create_file uploads the file as multipart form data and returns a BatchFile (id, filename, bytes, purpose, status, created_at). create_batch takes that file's id and returns a Batch. The optional metadata= is a dict of string keys and values returned on the batch object. The Batch API page has its size limits.

Wait for the batch

Python
batch.wait()   # blocks until terminal, refreshes the object in place

Batch.wait() polls while status is one of the four live values: validating, in_progress, finalizing or cancelling. It stops on the four terminal ones: completed, failed, expired or cancelled. Each poll updates the object in place, and wait() returns the batch itself. To classify a status yourself, import the two sets as ACTIVE_BATCH_STATUSES and TERMINAL_BATCH_STATUSES.

  • poll_interval= is the starting number of seconds between polls (default 5). It doubles each poll, up to 30 seconds, or up to your starting interval if you set one higher than 30. Zero or a negative value raises ValueError.
  • timeout= limits the wait, in seconds. None, the default, waits forever. When it runs out, wait() raises Python's built-in TimeoutError with the last status it saw in the message.
  • on_unknown_status= decides what happens if the server reports a status in neither set, which means your SDK is older than the platform. The default, "raise", raises UnknownStatusError (carrying .batch_id and .status). It raises because the batch may still be running, and treating it as finished could let results() return a partial file. Pass "return" and wait() returns with a warning.

A timeout stops only the waiting: the batch keeps running on the server. Wait again, or stop it with cancel_batch:

Python
try:
    batch.wait(timeout=3600)
except TimeoutError:
    print("still running:", batch.status, batch.request_counts)

Read the results

Python
summaries = {}
if batch.output_file_id:
    for row in batch.results():
        body = row["response"]["body"]
        summaries[row["custom_id"]] = body["choices"][0]["message"]["content"]

if batch.error_file_id:
    for row in batch.error_results():
        print("failed:", row["custom_id"], row["error"])

results() downloads the output file and yields one plain dict per JSONL line (custom_id, response, error), as written in the file. error_results() does the same for the error file, which holds the requests that failed. Output order is not guaranteed, so collect by custom_id. Both download the whole file before reading it, and a malformed line raises PrivatemindError.

What each raises:

  • Before the batch is terminal, both raise PrivatemindError telling you to wait() first.
  • A terminal batch with no output file means no request succeeded. If the whole input file was rejected, results() and error_results() both raise and point you to batch.errors, which says why. Otherwise results() raises and points you to error_results().
  • A terminal batch with no error file had no request fail, and error_results() yields nothing. On a cleanly completed batch, the if batch.error_file_id: check in the example above is optional.

To keep the raw file itself, download its bytes by id:

Python
raw = client.get_file_content(batch.output_file_id)
with open("batch-output.jsonl", "wb") as fh:
    fh.write(raw)

The batch lifecycle

A batch starts in_progress and ends in one of the four terminal statuses, named as in the Batch API:

  • completed: finished. output_file_id is set, and error_file_id too if any request errored.
  • failed: the input file was rejected as a whole; the diagnosis is on batch.errors.
  • expired: the 24 hour window ran out first. Finished requests are in the output file, the rest in the error file.
  • cancelled: you cancelled it. A batch with requests in flight finishes them first and shows cancelling meanwhile. Completed work is billed and stays readable in the output file.

Progress is in request_counts (total, completed, failed). Cancelled and expired requests count toward total only, never failed. Check it with get_batch:

Python
b = client.get_batch(batch.id)
print(b.status, b.request_counts.completed, "of", b.request_counts.total)

Pagination

list_batches returns one page as a plain list. It takes limit= (1 to 100) and after=, the id of the batch to start after. Pass after=page[-1].id for each next page until a page comes back empty:

Python
batches = []
page = client.list_batches(limit=100)
while page:
    batches.extend(page)
    page = client.list_batches(limit=100, after=page[-1].id)

The server can't filter by status, so filter in your code:

Python
in_progress = [b for b in batches if b.status == "in_progress"]

Called with neither limit= nor after=, list_batches() raises PrivatemindError if the server reports the list was truncated. The error tells you to pass after= for the next page. With either argument, the call returns the page without raising.

Handling errors

Every SDK exception subclasses PrivatemindError; the SDK overview has the hierarchy. The read calls (get_batch, list_batches, get_file_content and the polling inside wait()) retry temporary server errors on their own. The write calls (create_file, create_batch, cancel_batch) are never re-sent for you, so wrap them in your own retry where it matters:

Python
import time
from privatemind import RateLimitError

for attempt in range(5):
    try:
        batch = client.create_batch(
            input_file_id=f.id,
            endpoint="/v1/chat/completions",
            completion_window="24h",
        )
        break
    except RateLimitError as exc:
        time.sleep(exc.retry_after or 2 ** attempt)

RateLimitError.retry_after holds the server's Retry-After header, or None if it sent none.

Limits and retention

The server enforces the request count, file size and rate limits, so they can change without an SDK upgrade. A rejected request raises a normal SDK exception. See limits and retention for the current numbers.

Note

Batch input and output files hold your content and are kept on the server for up to 29 days. Batch is not eligible for zero-data-retention (ZDR) arrangements.

Where next

  • Batch API: the JSONL line format, the batch object, limits, and pricing.
  • Files: the endpoints behind create_file and get_file_content.
  • SDK overview: configuration, authentication, and the error hierarchy.