Data Movement · Topic 11 of 17
Kafka, Spark, Python and Native Connectors
Choose and configure the Snowflake connectors for Kafka, Spark and Python, understand their ingestion paths, and know what native connectors add.
- 5 min
- The Kafka connector writes
RECORD_CONTENTandRECORD_METADATAVARIANT columns, uses key pair authentication, and ingests with Snowpipe or Snowpipe Streaming. - Kafka classic v3 is planned for deprecation, and v4 is recommended for new work.
- The connector has no extra fee. Snowpipe or Snowpipe Streaming costs and storage apply.
What it is
Connectors link other platforms to Snowflake. Each one fits a different shape of work.
| Connector | Moves | Direction |
|---|---|---|
| Snowflake Connector for Kafka | Messages from Kafka topics | Into Snowflake |
| Snowflake Connector for Spark | DataFrames and query results | Both ways |
| Snowflake Connector for Python | SQL calls and result sets from Python code | Both ways |
| Native connectors | Data from specific apps and databases | Into Snowflake |
How it works
Kafka connector
The connector runs inside Kafka Connect and loads topic messages into Snowflake tables. It supports two ingestion methods, Snowpipe and Snowpipe Streaming. Snowflake describes the classic connector (v3) as fully supported but planned for future deprecation, and recommends the v4 connector for all new implementations.
- Mapping: topics map to tables. A topic that is not mapped gets a new table named after the topic, converted to a valid Snowflake identifier. One topic feeding one table is the normal pattern.
- Default schema: two VARIANT columns.
RECORD_CONTENTholds the message andRECORD_METADATAholds topic, partition, offset, timestamps, key and headers. - Snowpipe path: the connector buffers messages and writes them as files to an internal stage when a time, memory or message count threshold is reached. Snowpipe then loads the files, so latency depends on the flush settings.
- Snowpipe Streaming path: rows go straight to the table through channels without files, for lower latency.
- Authentication: key pair authentication for the Snowflake user the connector uses.
- Schema detection and evolution can be enabled so table columns follow the message structure.
- Fault tolerance: messages are not meant to be duplicated or silently dropped, but row order is not guaranteed to follow publish order. Single Message Transforms are not supported by the Snowflake converters, and each topic should be processed by only one connector instance.
There is no separate charge for the connector. You pay for Snowpipe processing or Snowpipe Streaming and for storage.
The comparison of Snowpipe Streaming and the Kafka connector is covered on the Snowpipe Streaming page. Here, remember that the connector is the packaged way to move Kafka topics, while Snowpipe Streaming is the lower level ingestion service it can use.
Spark connector
The Spark connector reads Snowflake tables into DataFrames and writes DataFrames back. It uses the source name net.snowflake.spark.snowflake.
df = (spark.read
.format("net.snowflake.spark.snowflake")
.options(**sf_options) # sfURL, sfUser, sfDatabase, sfSchema, sfWarehouse
.option("query", "SELECT * FROM orders WHERE order_date > '2026-01-01'")
.load())
df.write.format("net.snowflake.spark.snowflake") \
.options(**sf_options).option("dbtable", "orders_out").mode("append").save()
- Use
dbtablefor a whole table orqueryfor a SQL statement. A query is run by Snowflake, so filters and aggregates are computed in Snowflake and only the result reaches Spark. This is called query pushdown. - The connector moves data through a temporary stage, so writes use bulk loading rather than row by row inserts.
- Compute on the Snowflake side uses the virtual warehouse named in the options.
- A cloud storage location for the temporary stage can be supplied when needed.
Python connector
The Python connector (snowflake-connector-python) is a driver that implements the Python database API.
import snowflake.connector
con = snowflake.connector.connect(account="org-acct", user="svc_etl",
private_key_file="rsa_key.p8", warehouse="etl_wh", database="raw")
cur = con.cursor()
cur.execute("SELECT * FROM orders WHERE region = %s", ("EU",))
df = cur.fetch_pandas_all()
- Binding: use bind variables (
%sby default, orqmarkandnumericstyles) rather than string concatenation, which prevents SQL injection. - pandas:
fetch_pandas_all()andfetch_pandas_batches()need the pandas extra of the package.write_pandas()loads a DataFrame through a stage. - Async:
execute_async()starts a query without waiting; the query ID (cursor.sfqid) lets the program poll or fetch the results later. - Authentication: password, key pair, OAuth and browser based SSO are supported. Service accounts should use key pair or another non interactive method.
- Snowpark versus connector: the connector sends SQL text and returns result sets. Snowpark builds DataFrame operations that Snowflake executes as SQL on the server.
Native connectors
Native connectors are Snowflake built connectors that bring data from a specific source, such as a database or a SaaS application, into a Snowflake account. They are installed in the account, configured with source connection details and then ingest on a schedule or continuously, so there is no connector code to write.
Snowflake's newer integration service, Openflow, now offers many source specific connectors, including PostgreSQL, MySQL, SQL Server, Oracle, Salesforce and Kafka. For a current choice, check the connector list in the documentation, since the delivery method (Native App or Openflow) can differ by source.
Common mistakes
- Choosing the Snowpipe path of the Kafka connector and expecting second level latency. File buffering adds delay.
- Expecting row order from Kafka to be kept in the table.
- Reading a whole table into Spark and filtering there instead of passing a
querythat Snowflake filters first. - Building SQL strings with user input in the Python connector instead of binding variables.
- Using the Python connector to run heavy transformations in the client. Push the work to Snowflake with SQL or Snowpark.
What to remember for the exam
- The Kafka connector writes
RECORD_CONTENTandRECORD_METADATAVARIANT columns, uses key pair authentication, and ingests with Snowpipe or Snowpipe Streaming. - Kafka classic v3 is planned for deprecation, and v4 is recommended for new work.
- The connector has no extra fee. Snowpipe or Snowpipe Streaming costs and storage apply.
- Spark connector: format
net.snowflake.spark.snowflake,dbtableorquery, query pushdown, temporary stage for transfer. - Python connector:
connect, cursors, bind variables,fetch_pandas_all,write_pandas,execute_async. - Native connectors ingest from a named source with no code, and Openflow is the newer service for the same job.
Check yourself
Which two VARIANT columns does the Kafka connector create by default?
RECORD_CONTENTfor the message andRECORD_METADATAfor topic, partition, offset and other details.How does the Spark connector make sure filtering happens in Snowflake?
Pass a
queryoption, so Snowflake runs the SQL and only the result is returned to Spark (query pushdown).How should a Python program pass user supplied values into a query?
Use bind variables in
cursor.execute()instead of building the SQL string.
Topic 11 of 17 in Data Movement
Test yourself on Data Movement
The full Snowflake DEA-C02 question bank has 709 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Snowflake. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.