Implement Computer Vision Solutions · Topic 2 of 2
Multimodal Understanding and Visual Content Safety
AI-103 visual understanding: captions, alt text and visual Q&A with multimodal models, Content Understanding for images and video, and safety for visuals.
- 4 min
- Multimodal models: captions, alt text, visual Q&A, comparing images.
- Content Understanding analyzers give structured fields from images, video, audio and documents.
- Custom classification and detection with classic ML point to Azure Machine Learning AutoML; Custom Vision retires in 2028.
What it is
Multimodal understanding means getting meaning out of images and video: describing them, answering questions about them, and finding objects or regions. In Foundry you do this with multimodal models that accept images in the prompt, and with Azure Content Understanding in Foundry Tools, which turns visual content into structured results.
Visual content brings its own risks, so this skill group also covers responsible AI for images and video.
How it works
Using multimodal models
| Task | Approach |
|---|---|
| Captions | Ask for a short caption or a detailed description; specify length and audience |
| Alt text | Ask for concise alternative text plus an extended description, following accessibility guidance (describe what matters, not "image of") |
| Visual question answering | Send the image with a question; instruct the model to answer only from what is visible |
| Several images | Send multiple images in one request to compare or summarize them |
| Objects and regions | Ask the model, or use an analyzer, to identify objects and where they appear |
Images are passed in the message as a URL or as base64 data alongside the text.
Azure Content Understanding
Content Understanding uses analyzers to extract structured information from documents, images, audio and video.
- Prebuilt analyzers handle common content types.
- Custom analyzers define the fields you want, such as "product name" or "damage visible: yes or no".
- For video, it splits content into segments and can describe and extract fields per segment, such as a scene description or the actions shown.
- Standard mode is the default: one analyzer processes one input. Pro mode (introduced in a preview API, documents only) adds reasoning across multiple input documents and can use reference data to validate results.
Content Understanding is a good choice when the output must be consistent, structured fields rather than free text.
Other vision capabilities
Prebuilt image analysis in Azure Vision in Foundry Tools can return tags, detected objects and text (OCR) without prompting a model. It is still available, so you may see it in questions; generative and Content Understanding approaches are the main focus of AI-103.
Azure Custom Vision retires on September 25, 2028. For custom image classification or object detection models trained with classic machine learning, Microsoft's migration guidance points to Azure Machine Learning automated ML (AutoML); it also lists generative models in Foundry and Content Understanding classification as options.
Responsible AI for visual content
| Risk | Control |
|---|---|
| Unsafe or disallowed images | Image content filters that classify violence, sexual, hate and self-harm content |
| Indirect prompt injection through images | Text hidden in an image, such as "ignore your instructions", can steer the model; treat text in images as untrusted data, and screen extracted text with Prompt Shields for indirect attacks |
| Brand and policy rules | Check generated images for prohibited symbols and correct brand use; apply watermarks or content credentials to mark AI-generated media |
| Inappropriate generated content | Guardrails on generation models block unsafe prompts and outputs |
A worked example: a retailer lets customers upload photos of damaged items. A multimodal model describes the damage; a Content Understanding custom analyzer extracts "item type" and "damage severity"; image filters block unsafe uploads; and the instructions tell the model to ignore any text written in the photo.
When the requirement is consistent, structured fields from images or video, choose Content Understanding. When it is a flexible description or an answer to a question, a multimodal model fits.
How the exam asks about this
- "Generate alt text for product images that follows accessibility guidance." A multimodal model with alt text instructions.
- "Extract the same set of fields from each video segment." Content Understanding with a custom analyzer.
- "An uploaded image contains written instructions to reveal data." Indirect prompt injection; treat the text as untrusted and screen it with Prompt Shields.
What to remember for the exam
- Multimodal models: captions, alt text, visual Q&A, comparing images.
- Content Understanding analyzers give structured fields from images, video, audio and documents.
- Custom classification and detection with classic ML point to Azure Machine Learning AutoML; Custom Vision retires in 2028.
- Text inside images can be an indirect prompt injection.
- Use filters, policy checks, watermarks and content credentials for visual safety.
Do not trust text found in an image as an instruction. Treat it as data, like any other untrusted input.
Common mistakes
Asking for "image of" style alt text. Describe the meaningful content.
Using free text output when fields are needed. Use an analyzer with defined fields.
Recommending Custom Vision for a new custom model. Use AutoML in Azure Machine Learning, or a generative or Content Understanding approach.
Check yourself
When should you use Content Understanding instead of a free-form multimodal prompt?
When you need consistent, structured fields extracted from images, video, audio or documents.
How can an image carry a prompt injection?
It can contain written instructions that the model reads and might follow.
Which service does Microsoft suggest for training custom image classification models with classic machine learning now that Custom Vision is retiring?
Azure Machine Learning automated ML (AutoML).
Topic 2 of 2 in Implement Computer Vision Solutions
Test yourself on Implement Computer Vision Solutions
The full Microsoft AI-103 question bank has 648 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Microsoft. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.