Identify AI Workloads · Topic 2 of 3
Speech and Computer Vision Capabilities
AI-901 speech and vision: speech recognition and synthesis, real time and batch transcription, SSML, image analysis, OCR, multimodal models and image generation.
- 5 min
- Speech recognition turns speech into text. Speech synthesis turns text into speech.
- Real time is for live audio. Batch is for large sets of stored files.
- A phrase list biases recognition toward specific words without training.
What it is
Speech AI connects spoken language and text. Speech recognition, also called speech to text, turns what people say into written words. Speech synthesis, also called text to speech, turns written words into spoken audio.
Computer vision AI understands pictures. It can describe an image, read text in it, detect objects, or answer questions about it. Image generation goes the other way and creates a picture from a written description.
How it works
Speech recognition
Speech recognition produces a transcript from audio. The main ways to use it are:
| Mode | How it works | Good for |
|---|---|---|
| Real time transcription | Audio is processed as it is spoken and text appears at once | Live captions, voice commands |
| Fast transcription | A recorded file is transcribed quickly and synchronously | Short recordings |
| Batch transcription | Large sets of stored files are submitted and processed asynchronously | Thousands of recordings, no waiting on each result |
Recognition can be improved for your own vocabulary. A phrase list gives the recogniser a handful of words, such as brand names, that it should expect, without any training. A custom speech model is trained on your own audio and text when industry wording is more extensive.
Other capabilities include recognising several speakers, detecting the spoken language, and translating speech into another language.
Speech synthesis
Speech synthesis uses neural voices that sound natural. You choose a voice, such as an English or Spanish voice, and send text to be spoken. The result is audio you can play or save.
SSML, the Speech Synthesis Markup Language, gives finer control over how the text is spoken.
| SSML element | Effect |
|---|---|
voice | Chooses which voice speaks the text |
prosody | Adjusts pitch, speaking rate and volume |
break | Inserts a pause |
emphasis | Stresses a word |
phoneme | Spells out a pronunciation phonetically |
A custom lexicon fixes the pronunciation of specific words, for example a drug name. A speaking style in SSML makes narration sound cheerful or calm. A custom voice creates a unique voice for a brand, with strict controls.
Computer vision
Computer vision covers several tasks.
| Task | What it returns |
|---|---|
| Image analysis | A caption, tags and a description of what is in the picture |
| Object detection | The objects present and where they are, as boxes |
| Optical character recognition (OCR) | The text that appears in an image |
| Visual question answering | An answer to a question about an image |
| Video analysis | Scenes, transcripts and summaries of recorded video |
Two kinds of tool provide these abilities. A dedicated vision service is built for analysis tasks such as OCR and object detection and returns structured results. A multimodal model is a language model that also accepts images, so you send a picture and a question and it replies in text. A multimodal model reads text in photos and answers questions well, but it is not a detector, so it does not return pixel exact boxes for objects.
Image generation
Image generation models create new images from a text prompt. Many can also edit an existing image, for example by changing a region. Images produced by Azure models include provenance information that records that they were generated by AI. Content filters check both the prompt and the result.
Choosing the capability
| The need | Capability |
|---|---|
| Captions that appear within a second or two of a speaker | Real time speech recognition |
| Transcribe thousands of stored recordings without waiting | Batch transcription |
| A voice assistant that reads replies aloud | Speech synthesis |
| A few brand names are misheard in captions | A phrase list |
| Read the text on a street sign in a photo | OCR |
| Describe a product photo in a sentence | Image analysis or a multimodal model |
| Create a banner image from a description | Image generation |
A worked scenario
A conference app shows live captions of each talk. It needs low delay, so it uses real time speech recognition, with a phrase list for the speakers' names and product names. After the event, the recordings go through batch transcription so that a searchable archive can be built. The app also has an accessibility feature that reads session descriptions aloud with a neural voice, and uses SSML to slow the speaking rate. Attendees can photograph a slide and ask a multimodal model to explain it.
What to remember for the exam
- Speech recognition turns speech into text. Speech synthesis turns text into speech.
- Real time is for live audio. Batch is for large sets of stored files.
- A phrase list biases recognition toward specific words without training.
- SSML controls voice, pitch, rate, pauses and pronunciation.
- OCR reads text in images. A multimodal model answers questions about images but does not return exact object boxes.
"Within a second or two" means real time. "Thousands of stored files" means batch.
How the exam asks about this
- "Transcribe thousands of audio files without waiting on each result." Batch transcription.
- "A synthesised voice mispronounces a drug name." A custom lexicon.
- "Which SSML element adjusts pitch and speaking rate?" prosody.
- "Extract the text shown on slides in a video." OCR on the frames.
Common mistakes
Using batch transcription for live captions. It is asynchronous, so it cannot give captions within seconds.
Assuming a multimodal model replaces a detector. It describes and answers, but does not give pixel exact object boxes.
Training a model for a few brand names. A phrase list is simpler.
Thinking synthesis and recognition are the same service direction. One speaks, the other listens.
Check yourself
Which transcription mode suits live captions?
Real time transcription.
What does a phrase list do?
It biases recognition toward specific words, such as brand names, without training a model.
What does OCR extract?
The text that appears in an image or video frame.
Topic 2 of 3 in Identify AI Workloads
Test yourself on Identify AI Workloads
The full Microsoft AI-901 question bank has 511 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Microsoft. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.