Visual Input and Vision Applications

What it is

A multimodal model accepts more than text. When it accepts images, you can show it a picture and ask a question in plain language: describe this photo, read the text on this label, compare these two diagrams. The model answers in text.

A vision application wraps that capability in a small program, for example one that lets users upload a product photo and returns a written description.

How it works

Sending an image

The message content is a list, with a text part for the instruction and an image_url part for the picture.

messages = [
    {"role": "system", "content": "You describe product photos for a catalogue."},
    {"role": "user", "content": [
        {"type": "text", "text": "Describe this product in two sentences."},
        {"type": "image_url", "image_url": {"url": "https://example.com/photos/lamp.jpg", "detail": "auto"}},
    ]},
]

reply = openai.chat.completions.create(
    model="your-multimodal-deployment",
    messages=messages,
    max_tokens=300,
)
print(reply.choices[0].message.content)

The url can be:

• A web address the service can reach, such as a link to a stored image.
• A base64 data URL for a local file, written as data:image/jpeg;base64,<encoded data>. The app reads the file and encodes it first.

The detail setting

The optional detail value controls how closely the model looks at the image.

Images use tokens, so larger or higher detail images cost more and use more of the context window.

What the model does well

• Describing and summarising what is in a picture.
• Answering questions about an image.
• Reading text that appears in a photo, such as a sign or a slide.
• Comparing several images that you send in one message.

Limitations to plan for

For video, a chat call cannot take the file directly. A practical approach is to sample several frames from the clip and send them as separate image parts, together with a transcript of the audio as text.

Vision applications beyond the multimodal model

For tasks such as OCR, object detection or tagging with structured output, a dedicated vision service may be the better choice, because it returns exact coordinates and a fixed format. A multimodal model suits open ended description and questions. Many applications use both.

Handling errors and cost

A 429 response means the rate limit was reached, so the app waits and retries with backoff. Reducing detail to low and resizing very large images before sending them lowers cost. Dropping detail and resending at once does not fix a rate limit error.

A worked scenario

An online marketplace lets sellers upload a photo and receive a draft listing. The app encodes the photo as a base64 data URL, sends it with the instruction "Write a title and a two sentence description", and sets detail to auto. The model's reply becomes the draft. For photos of price labels with small print, the team switches to high detail. When the team asks for the exact position of each item in the photo, they find the multimodal model does not return boxes, so they add a vision service for object detection. Product videos are handled by sending a few sampled frames and the narration transcript.