Audio, Video and Extraction Applications

What it is

Content Understanding also analyzes audio and video. For audio it transcribes speech and summarises it. For video it splits the recording into meaningful segments and describes each one. You then build a small application that sends files to an analyzer and reads back the result.

How it works

Audio analysis

The prebuilt-audioSearch analyzer processes audio and video files with speech. It:

• Transcribes the conversation.
• Generates a one paragraph summary of it.
• Supports multiple locales, so content in different languages can be processed.

The prebuilt-callCenter analyzer applies the same ideas to call recordings, and extracts topics, sentiment and key insights. A custom audio analyzer can add your own fields, for example "call outcome" classified from a list, or "action items" generated from the conversation.

Video analysis

The prebuilt-videoSearch analyzer processes video. It:

• Segments the video into sections, based on topic changes, scene changes and visual cues.
• Extracts a transcript for each segment.
• Writes a description of each segment, focused on people, places and actions, and a summary.

Segmentation is controlled by the enableSegment setting of the analyzer. For a recorded training session, the result can include a transcript, a list of scenes and a summary of each, which makes long recordings searchable.

Audio and video compared

Calling the service from Python

The Python package is azure-ai-contentunderstanding. The client is ContentUnderstandingClient, created from the resource endpoint and a credential. The analyze call names the analyzer by its ID and gives the input file.

import os
from azure.ai.contentunderstanding import ContentUnderstandingClient
from azure.ai.contentunderstanding.models import AnalysisInput
from azure.core.credentials import AzureKeyCredential

client = ContentUnderstandingClient(
    endpoint=os.environ["CONTENTUNDERSTANDING_ENDPOINT"],
    credential=AzureKeyCredential(os.environ["CONTENTUNDERSTANDING_KEY"]),
)

poller = client.begin_analyze(
    analyzer_id="prebuilt-audioSearch",
    inputs=[AnalysisInput(url="https://example.com/calls/call-0142.wav")],
)
result = poller.result()

content = result.contents[0]
print(content.markdown)
for name, field in content.fields.items():
    print(name, field.value, field.confidence)

Analysis takes time, so the call returns a poller. Calling result() waits until the operation has finished.

The REST pattern

The same service can be called with REST. A POST request to the analyzer's analyze address starts the work, and the key is sent in the Ocp-Apim-Subscription-Key header.

POST {endpoint}/contentunderstanding/analyzers/{analyzer-id}:analyze?api-version=2025-11-01

The service replies with 202 Accepted and an Operation-Location header. The client then polls that address, at an interval of a second or two and not in a tight loop, until the status is Succeeded. The result holds contents, each with markdown and fields.

Asynchronous, not real time

Content Understanding processes files asynchronously, so it is not meant for live captions within a second or two. For that need, use Azure Speech real time transcription. Content Understanding suits recordings that are already stored.

Keeping the application safe

• Read the key and endpoint from environment variables or Azure Key Vault. Do not put them in source code, in a query string or in a file committed to a repository.
• Prefer Microsoft Entra ID with a managed identity where possible.
• Check each field's confidence score and the result status before using the data.
• Respect rate limits by waiting and retrying.

A worked scenario

A call centre stores thousands of recorded calls. A developer writes a Python script that loops over the files and sends each to prebuilt-callCenter with begin_analyze. The script waits for each poller, reads the summary, the topics and the sentiment, and saves them to a database. Calls where a custom "escalation" field was classified as true are flagged for a supervisor. The key is read from an environment variable. For a separate live event, the same company shows real time captions with Azure Speech, because the analysis service works on files after the fact.