A vision-capable model reads images sent with your text. Send the message content as an array of typed parts, OpenAI's standard multimodal format. The Messages API takes images and documents as content blocks: see Messages API below.
curl -s "https://api.privatemind.com/v1/chat/completions" \
-H "Authorization: Bearer $PMIND_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "vision",
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "What is in this image?"},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}}
]
}]
}'The content array can mix any number of text and image_url parts.
Image sources
- Data URL:
data:image/<mime>;base64,<payload>. Use this when you have the image bytes locally. - HTTPS URL: a public URL, which the model must be able to fetch. Data URLs are more reliable.
PNG and JPEG always work. WebP and GIF depend on the engine serving the model.
Encoding a local file
import base64
from openai import OpenAI
client = OpenAI(base_url="https://api.privatemind.com/v1", api_key="PMIND...:...")
with open("photo.png", "rb") as f:
b64 = base64.b64encode(f.read()).decode()
resp = client.chat.completions.create(
model="vision",
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "Describe this."},
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
],
}],
)
print(resp.choices[0].message.content)Which models are vision-capable?
A model that accepts images has image_input: true in its capabilities on GET /v1/models. Check that flag, not the model id.
Images use tokens: more for larger images, depending on how the model splits them into tiles. Resize before sending if you don't need full resolution.
Messages API
On POST /v1/messages, images and documents are content blocks in a user turn. Each block has a source:
| Block | source.type |
Notes |
|---|---|---|
image |
base64 |
media_type plus the base64 data |
image |
url |
A public URL the model can fetch |
image |
file |
A file_id from the Files API; PNG, JPEG, GIF or WebP |
document |
text |
Plain text in data, with an optional title |
document |
content |
An array of text blocks |
document |
file |
A file_id holding text |
import base64
import anthropic
client = anthropic.Anthropic(base_url="https://api.privatemind.com", api_key="PMIND...:...")
with open("photo.png", "rb") as f:
b64 = base64.b64encode(f.read()).decode()
message = client.messages.create(
model="vision",
max_tokens=1024,
messages=[{"role": "user", "content": [
{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": b64}},
{"type": "text", "text": "Describe this."},
]}],
)PDF documents are not supported and return 400. Extract the text and send it as a text document. An image inside a tool_result works too: the model sees it as part of the next user turn.
Where next
- Chat completions for the underlying request shape.
- Messages API for the Anthropic-shape request.
- Models to identify vision-capable models in your catalog.
- Tool use for combining vision with function calling.