Get API key

Vision

View as Markdown

Multimodal chat: text plus images, in the OpenAI content-array shape or as Messages API blocks.

A vision-capable model reads images sent with your text. Send the message content as an array of typed parts, OpenAI's standard multimodal format. The Messages API takes images and documents as content blocks: see Messages API below.

cURL
curl -s "https://api.privatemind.com/v1/chat/completions" \
  -H "Authorization: Bearer $PMIND_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vision",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "What is in this image?"},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}}
      ]
    }]
  }'

The content array can mix any number of text and image_url parts.

Image sources

  • Data URL: data:image/<mime>;base64,<payload>. Use this when you have the image bytes locally.
  • HTTPS URL: a public URL, which the model must be able to fetch. Data URLs are more reliable.

PNG and JPEG always work. WebP and GIF depend on the engine serving the model.

Encoding a local file

Python
import base64
from openai import OpenAI

client = OpenAI(base_url="https://api.privatemind.com/v1", api_key="PMIND...:...")

with open("photo.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

resp = client.chat.completions.create(
    model="vision",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this."},
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
        ],
    }],
)
print(resp.choices[0].message.content)

Which models are vision-capable?

A model that accepts images has image_input: true in its capabilities on GET /v1/models. Check that flag, not the model id.

Note

Images use tokens: more for larger images, depending on how the model splits them into tiles. Resize before sending if you don't need full resolution.

Messages API

On POST /v1/messages, images and documents are content blocks in a user turn. Each block has a source:

Block source.type Notes
image base64 media_type plus the base64 data
image url A public URL the model can fetch
image file A file_id from the Files API; PNG, JPEG, GIF or WebP
document text Plain text in data, with an optional title
document content An array of text blocks
document file A file_id holding text
Python
import base64
import anthropic

client = anthropic.Anthropic(base_url="https://api.privatemind.com", api_key="PMIND...:...")

with open("photo.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

message = client.messages.create(
    model="vision",
    max_tokens=1024,
    messages=[{"role": "user", "content": [
        {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}},
        {"type": "text", "text": "Describe this."},
    ]}],
)

PDF documents are not supported and return 400. Extract the text and send it as a text document. An image inside a tool_result works too: the model sees it as part of the next user turn.

Where next