Implement Text Analysis Solutions · Topic 2 of 2
Speech Solutions for Agents
AI-103 speech: speech to text and text to speech for agents, custom speech models, speech as an agent modality, audio reasoning and speech translation.
- 3 min
- Azure Speech: speech to text, text to speech, speech translation, custom speech.
- SSML controls voice, rate, pitch, pauses and pronunciation.
- Pipeline design (STT, agent, TTS) versus a real-time audio model.
What it is
Speech lets users talk to an agent and hear its answers. Azure Speech in Foundry Tools provides speech to text (recognition), text to speech (synthesis) and speech translation. Newer audio-capable models can also listen and reason over audio directly.
How it works
Core capabilities
| Capability | What it does | Example |
|---|---|---|
| Speech to text | Transcribes spoken audio, in real time or in batch | Live captions, call transcription |
| Text to speech | Turns text into natural-sounding speech with neural voices | Voice replies from an agent |
| Speech translation | Translates spoken audio into text or speech in another language | Multilingual support calls |
| Custom speech | Adapts recognition to your vocabulary, accents or noisy environments | Product names, medical terms |
Speech Synthesis Markup Language (SSML) controls how text is spoken: voice, speaking rate, pitch, pauses and pronunciation.
Speech as an agent modality
There are two common designs:
| Design | How it works | Trade-off |
|---|---|---|
| Pipeline | Speech to text, then the agent, then text to speech | Flexible and easy to inspect; adds some latency |
| Real-time audio model | A model takes audio in and returns audio out in one low-latency session | Most natural conversation; fewer separate steps to control |
For both designs, handle interruptions (the user starts speaking while the agent talks) and turn detection (knowing when the user has finished).
Custom speech models
Train a custom speech model when standard recognition makes errors with:
- domain vocabulary, such as product codes or medical terms,
- strong accents or specific speakers,
- background noise, such as a factory floor.
You provide sample audio with transcripts, or related text, to adapt the model, then deploy it to an endpoint your app uses.
Reasoning from audio
Multimodal models that accept audio can answer questions about a recording, such as "What did the caller agree to?", without a separate transcription step. Transcription first is still useful when you need searchable text, timestamps or an audit record.
Translating speech
- Speech translation in Azure Speech translates spoken input directly, with text or synthesized speech output.
- A pipeline can also transcribe, translate the text with a language model or Translator, then synthesize speech, which allows extra steps such as simplifying the wording.
A worked example: a support agent must take spoken questions in Spanish and answer in English for the staff member and Spanish for the caller. The app uses speech translation for the caller's audio, passes the text to the agent, and uses text to speech with a Spanish neural voice for the reply.
If recognition keeps getting product names wrong, the answer is a custom speech model (or adding phrases), not a bigger language model.
How the exam asks about this
- "Adjust pause length and pronunciation in an agent's spoken reply." Use SSML.
- "Transcription fails on technical terms." Train a custom speech model.
- "A natural, low-latency voice conversation with an agent." A real-time audio model session.
What to remember for the exam
- Azure Speech: speech to text, text to speech, speech translation, custom speech.
- SSML controls voice, rate, pitch, pauses and pronunciation.
- Pipeline design (STT, agent, TTS) versus a real-time audio model.
- Custom speech fixes vocabulary, accent and noise problems.
- Audio-capable models can reason over recordings directly.
Do not choose text to speech when the requirement is understanding what a caller said. That is speech to text, or an audio-capable model.
Common mistakes
Mixing up STT and TTS. Speech to text listens; text to speech speaks.
Ignoring interruptions in voice agents. Handle barge-in and turn detection.
Fixing vocabulary errors by changing the LLM. Customize the speech model.
Check yourself
What does SSML control?
How text is spoken, such as the voice, rate, pitch, pauses and pronunciation.
When should you train a custom speech model?
When standard recognition misses domain vocabulary, accents or speech in noisy environments.
What is the difference between a pipeline voice agent and a real-time audio model?
A pipeline chains speech to text, the agent and text to speech; a real-time model handles audio in and out in one low-latency session.
Topic 2 of 2 in Implement Text Analysis Solutions
Test yourself on Implement Text Analysis Solutions
The full Microsoft AI-103 question bank has 648 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Microsoft. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.