Spoken Prompts and Azure Speech Applications

What it is

There are two ways to build an application that listens and speaks. A multimodal model that accepts audio can take a recorded question and answer directly. The Azure Speech service, used through the Azure Speech SDK, converts speech to text and text to speech, and you connect it to any language model.

Which one you choose depends on whether you want one model to handle the whole exchange or separate, specialised steps.

How it works

Spoken prompts with a multimodal model

A model deployed with audio support accepts audio in a chat completions request as an input_audio content part. The part holds the encoded audio and its format, next to any text instruction.

import base64

with open("question.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode("utf-8")

messages = [
    {"role": "system", "content": "You are a helpful voice assistant. Keep answers short."},
    {"role": "user", "content": [
        {"type": "input_audio", "input_audio": {"data": audio_b64, "format": "wav"}},
    ]},
]

The roles are the same as for text. The system message carries standing instructions, the user message carries the recorded question, the assistant message carries earlier replies and a tool message carries function output.

To get a spoken answer, the request asks for audio output with a chosen voice. The reply then contains an audio object.

For continuous, low latency conversation, the Realtime API keeps a session open. Session properties include instructions, which are standing guidance, voice, which picks the voice for replies, input_audio_format, such as pcm16, which says how the audio you send is encoded, and temperature, which controls randomness.

The Azure Speech SDK

The Python package is azure-cognitiveservices-speech. You create a SpeechConfig with the key and endpoint of your Speech resource, read from environment variables.

Text to speech

import os
import azure.cognitiveservices.speech as speechsdk

speech_config = speechsdk.SpeechConfig(
    subscription=os.environ.get("SPEECH_KEY"),
    endpoint=os.environ.get("ENDPOINT"),
)
speech_config.speech_synthesis_voice_name = "en-US-Ava:DragonHDLatestNeural"

audio_config = speechsdk.audio.AudioOutputConfig(use_default_speaker=True)
synthesizer = speechsdk.SpeechSynthesizer(speech_config=speech_config, audio_config=audio_config)

result = synthesizer.speak_text_async("Your order has shipped.").get()
if result.reason == speechsdk.ResultReason.SynthesizingAudioCompleted:
    print("Done")

The property speech_synthesis_voice_name chooses the neural voice that speaks.

Speech to text

speech_config.speech_recognition_language = "en-US"
audio_config = speechsdk.audio.AudioConfig(use_default_microphone=True)
recognizer = speechsdk.SpeechRecognizer(speech_config=speech_config, audio_config=audio_config)

result = recognizer.recognize_once()
if result.reason == speechsdk.ResultReason.RecognizedSpeech:
    print(result.text)
elif result.reason == speechsdk.ResultReason.NoMatch:
    print("No speech could be recognised")

The property speech_recognition_language sets the language of the audio you expect.

For finer control of speech output, send SSML, with elements such as voice, prosody for pitch and rate, and break for pauses.

Choosing an approach

Errors you may meet

A 400 error can mean an unsupported audio format or a malformed body. A 401 means the key or credential is missing or wrong. A 404 means a deployment name that does not exist. A 429 means requests above the rate limit.

A worked scenario

A kiosk lets visitors ask questions aloud. The first version records each question to a WAV file and sends it as an input_audio part to a multimodal model, with a system message that says to answer briefly. The reply's transcript field is shown as captions on the screen. A second feature reads out the opening hours using the Speech SDK with a chosen neural voice, set through speech_synthesis_voice_name. Keys are read from environment variables, and when the kiosk misunderstands a brand name, the team adds it to a phrase list.