Data Engineering · Topic 10 of 16

Snowpark - Python, Scala and Java

Understand Snowpark DataFrames, lazy pushdown execution, UDFs and stored procedures, and how the Python, Scala and Java libraries differ for an Architect.

  • 4 min
In a nutshell
  • Snowpark has client libraries for Python, Scala and Java, and runs code in Snowflake through DataFrames, UDFs, UDTFs and stored procedures.
  • DataFrames are lazy; an action triggers execution and processing is pushed down as SQL.
  • Python gets the Anaconda channel and Snowpark pandas; Java and Scala use JARs on a stage.

What it is

Snowpark is a set of libraries and a server side runtime that let developers write data pipelines in Python, Scala or Java and run them inside Snowflake. The code builds DataFrames, which Snowflake turns into SQL, so data stays where it is and the work uses Snowflake compute.

The architectural point is simple: bring the code to the data, not the data to the code. That removes data movement, keeps governance in one place and avoids running a separate Spark cluster.

Client codePython, Scala or JavabuildsDataFramelazy plantranslated toSQLruns onWarehouseinside SnowflakereturnsResults

How it works

Core concepts

A session connects the client library to an account. A DataFrame represents a query, not data. Transformations such as select, filter, join and group_by only build a plan; nothing runs until an action such as collect, show, to_pandas or save_as_table asks for results.

This is lazy evaluation, and it lets Snowflake optimize the whole chain as one query. Because processing is pushed down, a DataFrame over a billion rows does not pull a billion rows to the client.

Beyond DataFrames, Snowpark lets you register code that runs on the server:

ObjectWhat it does
UDFScalar function written in the client language, used in SQL
UDTFTable function that returns rows
UDAFUser defined aggregate function (Python)
Stored procedureProcedural logic, with Snowpark DataFrame code inside, called with CALL

This code runs in a secure sandbox on virtual warehouse compute. Dependencies (JAR files or packages) are uploaded to a stage and referenced with imports or packages.

Snowpark Python

Python is the most widely used language. A session is created with Session.builder.configs(params).create(). The Python library offers the DataFrame API, Python UDFs, UDTFs, UDAFs and stored procedures.

Third party packages come from the Snowflake Anaconda channel, and packages from other sources can be added through supported package repositories. Snowpark pandas (the pandas API on Snowflake) runs familiar pandas code as SQL inside Snowflake.

Machine learning training can need a lot of memory, so Snowpark Python code can run on Snowpark-optimized warehouses, which give much more memory per node than a standard warehouse.

from snowflake.snowpark import Session
from snowflake.snowpark.functions import col

session = Session.builder.configs(params).create()
df = (session.table("orders")
        .filter(col("amount") > 100)
        .group_by("region")
        .agg({"amount": "sum"}))
df.write.save_as_table("region_totals")

Snowpark Scala and Java

Scala and Java share the same ideas as Python: a session, lazy DataFrames, and UDFs and stored procedures that run in Snowflake. Both run on the JVM, so server side code is packaged as JAR files that you upload to a stage and name in the IMPORTS clause.

Choose Scala or Java when the team already has JVM code or libraries to reuse, or needs strong typing. The execution model is the same, and DataFrame code reads much like the Python version.

AspectPythonScalaJava
DataFrame APIYesYesYes
Server side UDF, UDTF, procedureYesYesYes
Third party librariesAnaconda channel and supported repositoriesJARs on a stageJARs on a stage
Pandas APIYes (Snowpark pandas)NoNo
Typical teamData science, data engineeringJVM and Spark migrationJava developers

How Snowpark compares with other options

OptionWhere code runsData movement
SnowparkInside SnowflakeNone
Spark connectorIn a Spark clusterData leaves Snowflake except pushdown
Python connectorOn the clientResult sets only
SQLInside SnowflakeNone
Exam tip

A scenario that says "move a Spark job to Snowflake to avoid copying data out and managing a cluster" points to Snowpark. One that says "ML training needs more memory" points to Snowpark-optimized warehouses.

Common mistakes

Expecting a DataFrame to hold data. It is a lazy query plan. Calling collect or to_pandas on a huge table pulls everything to the client.

Treating Snowpark as only a client library. Its value is server side execution of UDFs and procedures inside Snowflake.

Forgetting stage imports for Java and Scala code. Server side JVM code needs its JAR on a stage and an IMPORTS reference.

Using a standard warehouse for memory hungry Python. Heavy ML or large in memory work belongs on a Snowpark-optimized warehouse.

What to remember for the exam

  • Snowpark has client libraries for Python, Scala and Java, and runs code in Snowflake through DataFrames, UDFs, UDTFs and stored procedures.
  • DataFrames are lazy; an action triggers execution and processing is pushed down as SQL.
  • Python gets the Anaconda channel and Snowpark pandas; Java and Scala use JARs on a stage.
  • Snowpark-optimized warehouses give more memory for large Python and ML jobs.
  • Snowpark removes data movement compared with Spark or client side processing.

Check yourself

  • What does a Snowpark DataFrame transformation such as filter do immediately?

    Nothing runs yet; it adds to a lazy plan that executes as SQL when an action is called.

  • Where does a Snowpark UDF execute?

    Inside Snowflake, on warehouse compute, in a secure sandbox.

  • Which warehouse type suits memory intensive Python ML training?

    A Snowpark-optimized warehouse.

Topic 10 of 16 in Data Engineering

Test yourself on Data Engineering

The full Snowflake ARA-C01 question bank has 687 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Snowflake. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.