Mixed Understanding

OpenAI Chat format messages array. messages[*].content may be a string or an array; array element type ∈ {text, image_url, video_url, audio_url, file_url}. A type the model does not support returns 422 model_not_support_capability.

Typical content shapes:

1. Plain text (string content, simplest form):

[{"role": "user", "content": "count 1 to 3"}]

2. Plain text (array content, the unified format when mixing with multimodal):

[{"role": "user", "content": [{"type": "text", "text": "count 1 to 3"}]}]

3. text + image:

[{"role": "user", "content": [
  {"type": "text", "text": "describe this image"},
  {"type": "image_url", "image_url": {"url": "https://example.com/x.png"}}
]}]

4. text + video:

[{"role": "user", "content": [
  {"type": "text", "text": "summarize this video"},
  {"type": "video_url", "video_url": {"url": "https://example.com/clip.mp4"}}
]}]

5. text + file:

[{"role": "user", "content": [
  {"type": "text", "text": "extract key points"},
  {"type": "file_url", "file_url": {"url": "https://example.com/doc.pdf"}}
]}]

6. Multi-turn conversation (system + multiple user/assistant turns):

[
  {"role": "system", "content": "You are a terse assistant."},
  {"role": "user", "content": "1+1?"},
  {"role": "assistant", "content": "2"},
  {"role": "user", "content": "3+3?"}
]

7. Mixed attachments of all types (a single user message containing 2 each of image_url / video_url / audio_url / file_url):

[{"role": "user", "content": [
  {"type": "text", "text": "Summarize the key information from the following images, videos, audio, and documents"},
  {"type": "image_url", "image_url": {"url": "https://example.com/image-1.png"}},
  {"type": "image_url", "image_url": {"url": "https://example.com/image-2.jpg"}},
  {"type": "video_url", "video_url": {"url": "https://example.com/clip-1.mp4"}},
  {"type": "video_url", "video_url": {"url": "https://example.com/clip-2.mp4"}},
  {"type": "audio_url", "audio_url": {"url": "https://example.com/audio-1.mp3"}},
  {"type": "audio_url", "audio_url": {"url": "https://example.com/audio-2.wav"}},
  {"type": "file_url", "file_url": {"url": "https://example.com/doc-1.pdf"}},
  {"type": "file_url", "file_url": {"url": "https://example.com/doc-2.docx"}}
]}]

Inline base64 (asynchronous mode only, when sync is omitted or false): In image_url / video_url / audio_url / file_url blocks, inline base64 in the {"url": ...} object form (data: URI, or at least 4096 encoded characters whose decoded content matches a known media signature) is uploaded to file storage and replaced with a URL before the request is persisted or submitted upstream; the original base64 is not stored. Shorter raw base64 or content with an unrecognized signature is passed through unchanged and is not size-validated. Limits apply only to inline base64 items (public HTTP(S) URLs are not limited): decoded size ≤ 5 MB per item, ≤ 5 inline items per array, ≤ 10 MB total per request, and ≤ 20 inline items per request. An invalid data: prefix or payload that decodes to zero bytes returns 422; exceeding a limit also returns 422. Use /v1/files/upload first and pass the resulting URL instead.

When sync: true, the request is not stored as a task and the rules above do not apply: inline base64 is submitted upstream unchanged without size or count validation.

Shapes outside the coverage scope: input_audio.data / file.file_data, and the direct string form {"type":"image_url","image_url":"data:..."} (where the value is not a {"url": ...} object). Inline base64 in these shapes is passed through unchanged.

Fixed at null during submit; obtained via GET /v1/tasks/{task_id} after the task completes — results[0] is the full OpenAI ChatCompletion response.

Known limitation: a thinking model's reasoning content (reasoning_content) appears only in the SSE stream's delta and is not accumulated into results[0].message.content