Snowflake Gen AI Functions · Topic 12 of 26
Text Splitting - SPLIT_TEXT_RECURSIVE_CHARACTER and SPLIT_TEXT_MARKDOWN_HEADER
Chunk text for embeddings and Cortex Search with SPLIT_TEXT_RECURSIVE_CHARACTER and SPLIT_TEXT_MARKDOWN_HEADER. Know arguments, defaults and return shapes.
- 4 min
- Both functions live in
SNOWFLAKE.CORTEXand return arrays. SPLIT_TEXT_RECURSIVE_CHARACTER(text, format, chunk_size, overlap, separators)hasformatofnoneormarkdown.- Default separators: paragraph break, line break, space, empty string.
What it is
A model or an embedding has a limited input size, and search works better on focused passages than on whole documents. Chunking cuts long text into shorter pieces first. Snowflake offers two SQL helper functions in the SNOWFLAKE.CORTEX schema for this.
SPLIT_TEXT_RECURSIVE_CHARACTERsplits any text by trying a list of separators in order.SPLIT_TEXT_MARKDOWN_HEADERsplits Markdown at its headers and tags every chunk with the headers above it.
Both return an array, which you turn into one row per chunk with LATERAL FLATTEN. The usual place to run them is between parsing and embedding.
How it works
SPLIT_TEXT_RECURSIVE_CHARACTER
SNOWFLAKE.CORTEX.SPLIT_TEXT_RECURSIVE_CHARACTER(
<text_to_split>, <format>, <chunk_size> [, <overlap>] [, <separators>])
| Argument | Meaning |
|---|---|
text_to_split | The text to cut |
format | none or markdown; decides the default separators |
chunk_size | Maximum characters per chunk, greater than zero |
overlap | Characters shared between consecutive chunks; default is none; must be smaller than chunk_size |
separators | Ordered list of split boundaries; default is paragraph break, line break, space, then any two characters |
With markdown, the function also splits on headers, code blocks and tables, in addition to any separators you give. With none, only the separators apply. The last separator should be a general one, such as an empty string, so the text can always be cut to size.
The function tries the first separator, and any chunk still longer than chunk_size is split again with the next one, repeating until every chunk fits. That is why it is called recursive.
SELECT SNOWFLAKE.CORTEX.SPLIT_TEXT_RECURSIVE_CHARACTER(
'hello world are you here', 'none', 15, 10);
-- ['hello world are', 'world are you', 'are you here']
Chunk size and overlap are measured in characters, not tokens. Overlap lets each chunk keep some context from the previous one.
To process a table, flatten the array.
SELECT d.doc_id, c.value::STRING AS chunk
FROM docs d,
LATERAL FLATTEN(input => SNOWFLAKE.CORTEX.SPLIT_TEXT_RECURSIVE_CHARACTER(
d.text, 'markdown', 1500, 150)) c;
Overlap repeats text in neighboring chunks, so more overlap means more rows and more tokens to embed. Use just enough to keep context across a border.
SPLIT_TEXT_MARKDOWN_HEADER
SNOWFLAKE.CORTEX.SPLIT_TEXT_MARKDOWN_HEADER(
<text_to_split>, <headers_to_split_on>, <chunk_size> [, <overlap>])
| Argument | Meaning |
|---|---|
text_to_split | A Markdown formatted string |
headers_to_split_on | A map of header markers to metadata names, for example OBJECT_CONSTRUCT('#', 'header_1', '##', 'header_2') |
chunk_size | Maximum characters per chunk, greater than zero |
overlap | Characters shared between chunks; default 0 |
The function first splits at the headers you list, then splits each section recursively with the default plain text separators until chunks fit chunk_size. It returns an array of objects, each with:
chunk: the extracted text.headers: the header values above the chunk, with keys named as in your map.
Header lines do not appear in the chunk text. They live only in the headers metadata. A chunk under only a level 1 header has just that key, and a chunk under a level 2 header has both.
SELECT d.doc_id,
c.value['chunk']::STRING AS chunk,
c.value['headers']['header_1']::STRING AS section
FROM docs d,
LATERAL FLATTEN(input => SNOWFLAKE.CORTEX.SPLIT_TEXT_MARKDOWN_HEADER(
d.markdown_text,
OBJECT_CONSTRUCT('#', 'header_1', '##', 'header_2'),
2000, 100)) c;
Choosing
| Situation | Function |
|---|---|
| Plain text, or you want your own separators | SPLIT_TEXT_RECURSIVE_CHARACTER with none |
Markdown from AI_PARSE_DOCUMENT in LAYOUT mode, headers matter for retrieval | SPLIT_TEXT_MARKDOWN_HEADER |
| Markdown, but you only need size limits and structure aware cutting | SPLIT_TEXT_RECURSIVE_CHARACTER with markdown |
Text too long for AI_REDACT | SPLIT_TEXT_RECURSIVE_CHARACTER |
Storing the header metadata beside each chunk lets a search result say which section it came from, and can be used as an attribute for filtering.
chunk_size is characters. Do not assume it means tokens, and leave room under the token limit of your embedding model.
Common mistakes
Setting overlap equal to or larger than chunk_size. It must be smaller.
Passing a format other than none or markdown. Only those two are allowed.
Reading the result as rows. It is an array; use LATERAL FLATTEN.
Expecting headers inside the chunk text from the markdown header splitter. They are in the headers field only.
Treating chunk_size as tokens. It counts characters.
What to remember for the exam
- Both functions live in
SNOWFLAKE.CORTEXand return arrays. SPLIT_TEXT_RECURSIVE_CHARACTER(text, format, chunk_size, overlap, separators)hasformatofnoneormarkdown.- Default separators: paragraph break, line break, space, empty string.
SPLIT_TEXT_MARKDOWN_HEADERreturnschunkandheadersfor each piece.- Sizes are measured in characters; overlap defaults to 0 or none.
- Use
LATERAL FLATTENto get one row per chunk.
Check yourself
What unit is
chunk_sizemeasured in?Characters.
Which splitter keeps the section titles as metadata?
SPLIT_TEXT_MARKDOWN_HEADER, in theheadersfield of each result object.What must be true of
overlap?It must be smaller than
chunk_size.
Topic 12 of 26 in Snowflake Gen AI Functions
Test yourself on Snowflake Gen AI Functions
The full Snowflake GES-C02 question bank has 585 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Snowflake. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.