Snowflake Gen AI Functions · Topic 12 of 26

Text Splitting - SPLIT_TEXT_RECURSIVE_CHARACTER and SPLIT_TEXT_MARKDOWN_HEADER

Chunk text for embeddings and Cortex Search with SPLIT_TEXT_RECURSIVE_CHARACTER and SPLIT_TEXT_MARKDOWN_HEADER. Know arguments, defaults and return shapes.

  • 4 min
In a nutshell
  • Both functions live in SNOWFLAKE.CORTEX and return arrays.
  • SPLIT_TEXT_RECURSIVE_CHARACTER(text, format, chunk_size, overlap, separators) has format of none or markdown.
  • Default separators: paragraph break, line break, space, empty string.

What it is

A model or an embedding has a limited input size, and search works better on focused passages than on whole documents. Chunking cuts long text into shorter pieces first. Snowflake offers two SQL helper functions in the SNOWFLAKE.CORTEX schema for this.

  • SPLIT_TEXT_RECURSIVE_CHARACTER splits any text by trying a list of separators in order.
  • SPLIT_TEXT_MARKDOWN_HEADER splits Markdown at its headers and tags every chunk with the headers above it.

Both return an array, which you turn into one row per chunk with LATERAL FLATTEN. The usual place to run them is between parsing and embedding.

StagePDF filesparseAI_PARSE_DOCUMENTLAYOUT modesplitSPLIT_TEXT functionchunksflattenChunk tableone row per chunkindexCortex Searchor AI_EMBED

How it works

SPLIT_TEXT_RECURSIVE_CHARACTER

SNOWFLAKE.CORTEX.SPLIT_TEXT_RECURSIVE_CHARACTER(
  <text_to_split>, <format>, <chunk_size> [, <overlap>] [, <separators>])
ArgumentMeaning
text_to_splitThe text to cut
formatnone or markdown; decides the default separators
chunk_sizeMaximum characters per chunk, greater than zero
overlapCharacters shared between consecutive chunks; default is none; must be smaller than chunk_size
separatorsOrdered list of split boundaries; default is paragraph break, line break, space, then any two characters

With markdown, the function also splits on headers, code blocks and tables, in addition to any separators you give. With none, only the separators apply. The last separator should be a general one, such as an empty string, so the text can always be cut to size.

The function tries the first separator, and any chunk still longer than chunk_size is split again with the next one, repeating until every chunk fits. That is why it is called recursive.

SELECT SNOWFLAKE.CORTEX.SPLIT_TEXT_RECURSIVE_CHARACTER(
  'hello world are you here', 'none', 15, 10);
-- ['hello world are', 'world are you', 'are you here']

Chunk size and overlap are measured in characters, not tokens. Overlap lets each chunk keep some context from the previous one.

To process a table, flatten the array.

SELECT d.doc_id, c.value::STRING AS chunk
FROM docs d,
     LATERAL FLATTEN(input => SNOWFLAKE.CORTEX.SPLIT_TEXT_RECURSIVE_CHARACTER(
                                d.text, 'markdown', 1500, 150)) c;
Exam tip

Overlap repeats text in neighboring chunks, so more overlap means more rows and more tokens to embed. Use just enough to keep context across a border.

SPLIT_TEXT_MARKDOWN_HEADER

SNOWFLAKE.CORTEX.SPLIT_TEXT_MARKDOWN_HEADER(
  <text_to_split>, <headers_to_split_on>, <chunk_size> [, <overlap>])
ArgumentMeaning
text_to_splitA Markdown formatted string
headers_to_split_onA map of header markers to metadata names, for example OBJECT_CONSTRUCT('#', 'header_1', '##', 'header_2')
chunk_sizeMaximum characters per chunk, greater than zero
overlapCharacters shared between chunks; default 0

The function first splits at the headers you list, then splits each section recursively with the default plain text separators until chunks fit chunk_size. It returns an array of objects, each with:

  • chunk: the extracted text.
  • headers: the header values above the chunk, with keys named as in your map.

Header lines do not appear in the chunk text. They live only in the headers metadata. A chunk under only a level 1 header has just that key, and a chunk under a level 2 header has both.

SELECT d.doc_id,
       c.value['chunk']::STRING AS chunk,
       c.value['headers']['header_1']::STRING AS section
FROM docs d,
     LATERAL FLATTEN(input => SNOWFLAKE.CORTEX.SPLIT_TEXT_MARKDOWN_HEADER(
         d.markdown_text,
         OBJECT_CONSTRUCT('#', 'header_1', '##', 'header_2'),
         2000, 100)) c;

Choosing

SituationFunction
Plain text, or you want your own separatorsSPLIT_TEXT_RECURSIVE_CHARACTER with none
Markdown from AI_PARSE_DOCUMENT in LAYOUT mode, headers matter for retrievalSPLIT_TEXT_MARKDOWN_HEADER
Markdown, but you only need size limits and structure aware cuttingSPLIT_TEXT_RECURSIVE_CHARACTER with markdown
Text too long for AI_REDACTSPLIT_TEXT_RECURSIVE_CHARACTER

Storing the header metadata beside each chunk lets a search result say which section it came from, and can be used as an attribute for filtering.

Watch out

chunk_size is characters. Do not assume it means tokens, and leave room under the token limit of your embedding model.

Common mistakes

Setting overlap equal to or larger than chunk_size. It must be smaller.

Passing a format other than none or markdown. Only those two are allowed.

Reading the result as rows. It is an array; use LATERAL FLATTEN.

Expecting headers inside the chunk text from the markdown header splitter. They are in the headers field only.

Treating chunk_size as tokens. It counts characters.

What to remember for the exam

  • Both functions live in SNOWFLAKE.CORTEX and return arrays.
  • SPLIT_TEXT_RECURSIVE_CHARACTER(text, format, chunk_size, overlap, separators) has format of none or markdown.
  • Default separators: paragraph break, line break, space, empty string.
  • SPLIT_TEXT_MARKDOWN_HEADER returns chunk and headers for each piece.
  • Sizes are measured in characters; overlap defaults to 0 or none.
  • Use LATERAL FLATTEN to get one row per chunk.

Check yourself

  • What unit is chunk_size measured in?

    Characters.

  • Which splitter keeps the section titles as metadata?

    SPLIT_TEXT_MARKDOWN_HEADER, in the headers field of each result object.

  • What must be true of overlap?

    It must be smaller than chunk_size.

Topic 12 of 26 in Snowflake Gen AI Functions

Test yourself on Snowflake Gen AI Functions

The full Snowflake GES-C02 question bank has 585 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Snowflake. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.