Snowflake Gen AI Functions · Topic 17 of 26

Multi-modal Analytics: Audio and Image Processing with Cortex AI Functions

Analyze audio and images in Snowflake: transcribe with AI_TRANSCRIBE, describe or classify images with AI_COMPLETE, and embed images for similarity.

  • 4 min
In a nutshell
  • Audio and video: AI_TRANSCRIBE. Images with reasoning: AI_COMPLETE with a vision model and PROMPT.
  • timestamp_granularity values: word and speaker.
  • Limits: 700 MB, 60 minutes with timestamps, 120 minutes without.

What it is

Multi-modal analytics means working with more than text: audio, video, images and documents. Snowflake handles each type with a function that fits it, and files are always referenced with TO_FILE from a stage.

There is no single function for all media. The exam asks which function takes which input.

InputFunctionWhat you get
Audio or video fileAI_TRANSCRIBETranscript, optional timestamps and speaker labels
ImageAI_COMPLETE with a vision modelDescription, classification, answers about the image
ImageAI_CLASSIFY, AI_FILTERLabel or yes/no on images
ImageAI_EMBED, AI_SIMILARITYVector or similarity score
Scanned page or image of textAI_PARSE_DOCUMENT, AI_EXTRACTText or named fields
Stageaudio filesTO_FILEAI_TRANSCRIBEtext and segmentspromptAI_COMPLETEsummary, sentimentStageimagesTO_FILEAI_COMPLETEvision modellabelTableresults

How it works

Audio with AI_TRANSCRIBE

AI_TRANSCRIBE(<audio_file> [, <options>]) takes a FILE object. It returns a VARCHAR holding JSON.

  • Audio formats: AAC, FLAC, M4A, MP3, MP4, OGG, WAV, WEBM. Video formats such as MP4, MOV and MKV work if there is an audio track.
  • Limits: 700 MB per file. Maximum duration is 60 minutes when you ask for word or speaker timestamps, and 120 minutes without timestamps.
  • Language is detected automatically, and speech must start within the first five seconds.
  • The option timestamp_granularity accepts 'word' (a timestamp per word) or 'speaker' (a timestamp and speaker label per turn, such as SPEAKER_00). Without it you get one text field for the whole file.
SELECT AI_TRANSCRIBE(TO_FILE('@calls', 'call_001.mp3'),
                     {'timestamp_granularity': 'speaker'}) AS transcript;

The output has audio_duration and either text or segments (each with start, end, text and, in speaker mode, speaker_label). Speaker diarization is therefore turned on through the 'speaker' granularity, not through a separate flag.

A common call center pipeline transcribes the call, then runs AI_SENTIMENT, AI_SUMMARIZE or AI_COMPLETE on the text. Transcription alone does not analyze meaning.

Exam tip

"Speaker labels in a call recording" means AI_TRANSCRIBE with timestamp_granularity set to speaker.

Images with AI_COMPLETE

Images go into a prompt object with PROMPT and TO_FILE. Numbered placeholders match the files.

SELECT AI_COMPLETE('claude-sonnet-4-5',
  PROMPT('Is there a damaged package in image {0}? Answer yes or no.',
         TO_FILE('@photos', 'dock_17.png'))) AS verdict;

Rules from the docs for this variant:

  • Formats: .jpg, .jpeg, .png, .gif, .webp.
  • Maximum image size is 10 MB for most models and 3.75 MB for Claude models.
  • Only some models accept images, so choose a vision capable one.
  • The stage must use server side encryption.
  • AI_COMPLETE does not accept audio or video.
  • Apply the function to a column of FILE objects, for example from a directory table, to process a whole stage as a batch.

For simple labeling, AI_CLASSIFY accepts text or images and returns the matching category, which is cheaper to reason about than free text.

Image similarity and embeddings

AI_EMBED with the model voyage-multimodal-3 turns an image (JPEG, WEBP, PNG or BMP, up to 10 MB) into a VECTOR. AI_SIMILARITY does the comparison directly and defaults to voyage-multimodal-3 for images. Both inputs must be images. Text to image comparison is not supported by AI_SIMILARITY.

Putting it together

A support team that receives voice messages and photos of damaged goods needs both paths. Voice goes through AI_TRANSCRIBE and then a text function. Photos go through a vision model. Both results land in one table, where AI_COMPLETE or AI_CLASSIFY can combine the two into a final label such as "refund" or "replace". Each step is a separate function call, and the files stay on the stage the whole time.

Regions and access

AI_TRANSCRIBE is listed in a small set of regions (AWS US West 2, AWS US East 1, AWS EU Central 1 and Azure East US 2), and image support in AI_EMBED is also region limited. Check availability before a design depends on it. Callers need the USE AI FUNCTIONS privilege and the SNOWFLAKE.CORTEX_USER database role.

Common mistakes

Sending audio to AI_COMPLETE. It will not process audio. Transcribe first.

Expecting AI_TRANSCRIBE to summarize. It returns text only.

Asking for word timestamps on a 90 minute file. With timestamps the limit is 60 minutes.

Using a text only model for an image prompt. Only the listed vision capable models accept image files.

Using a stage with client side encryption. FILE based functions do not support it.

Watch out

Images in a user stage or table stage cannot be used as FILE values for AI functions. Use a named stage with server side encryption.

What to remember for the exam

  • Audio and video: AI_TRANSCRIBE. Images with reasoning: AI_COMPLETE with a vision model and PROMPT.
  • timestamp_granularity values: word and speaker.
  • Limits: 700 MB, 60 minutes with timestamps, 120 minutes without.
  • Images are compared only with images; use voyage-multimodal-3.
  • Transcribe first, then analyze the text.

Check yourself

  • Which function converts a customer call recording into text with speaker turns?

    AI_TRANSCRIBE with timestamp_granularity set to speaker.

  • Can AI_COMPLETE take an MP3 directly?

    No, it accepts text, images and documents, so transcribe the audio first.

  • Which model embeds images for AI_SIMILARITY?

    voyage-multimodal-3.

Topic 17 of 26 in Snowflake Gen AI Functions

Test yourself on Snowflake Gen AI Functions

The full Snowflake GES-C02 question bank has 585 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Snowflake. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.