Data Movement · Topic 11 of 17

Kafka, Spark, Python and Native Connectors

Choose and configure the Snowflake connectors for Kafka, Spark and Python, understand their ingestion paths, and know what native connectors add.

  • 5 min
In a nutshell
  • The Kafka connector writes RECORD_CONTENT and RECORD_METADATA VARIANT columns, uses key pair authentication, and ingests with Snowpipe or Snowpipe Streaming.
  • Kafka classic v3 is planned for deprecation, and v4 is recommended for new work.
  • The connector has no extra fee. Snowpipe or Snowpipe Streaming costs and storage apply.

What it is

Connectors link other platforms to Snowflake. Each one fits a different shape of work.

ConnectorMovesDirection
Snowflake Connector for KafkaMessages from Kafka topicsInto Snowflake
Snowflake Connector for SparkDataFrames and query resultsBoth ways
Snowflake Connector for PythonSQL calls and result sets from Python codeBoth ways
Native connectorsData from specific apps and databasesInto Snowflake
Kafka topicKafka Connect workerKafka connectorbuffer, then loadSnowpipe or Snowpipe StreamingTableSpark jobDataFrame read or writeSpark connectortemp stage transferCOPY and unloadTable

How it works

Kafka connector

The connector runs inside Kafka Connect and loads topic messages into Snowflake tables. It supports two ingestion methods, Snowpipe and Snowpipe Streaming. Snowflake describes the classic connector (v3) as fully supported but planned for future deprecation, and recommends the v4 connector for all new implementations.

  • Mapping: topics map to tables. A topic that is not mapped gets a new table named after the topic, converted to a valid Snowflake identifier. One topic feeding one table is the normal pattern.
  • Default schema: two VARIANT columns. RECORD_CONTENT holds the message and RECORD_METADATA holds topic, partition, offset, timestamps, key and headers.
  • Snowpipe path: the connector buffers messages and writes them as files to an internal stage when a time, memory or message count threshold is reached. Snowpipe then loads the files, so latency depends on the flush settings.
  • Snowpipe Streaming path: rows go straight to the table through channels without files, for lower latency.
  • Authentication: key pair authentication for the Snowflake user the connector uses.
  • Schema detection and evolution can be enabled so table columns follow the message structure.
  • Fault tolerance: messages are not meant to be duplicated or silently dropped, but row order is not guaranteed to follow publish order. Single Message Transforms are not supported by the Snowflake converters, and each topic should be processed by only one connector instance.
Exam tip

There is no separate charge for the connector. You pay for Snowpipe processing or Snowpipe Streaming and for storage.

The comparison of Snowpipe Streaming and the Kafka connector is covered on the Snowpipe Streaming page. Here, remember that the connector is the packaged way to move Kafka topics, while Snowpipe Streaming is the lower level ingestion service it can use.

Spark connector

The Spark connector reads Snowflake tables into DataFrames and writes DataFrames back. It uses the source name net.snowflake.spark.snowflake.

df = (spark.read
  .format("net.snowflake.spark.snowflake")
  .options(**sf_options)        # sfURL, sfUser, sfDatabase, sfSchema, sfWarehouse
  .option("query", "SELECT * FROM orders WHERE order_date > '2026-01-01'")
  .load())

df.write.format("net.snowflake.spark.snowflake") \
  .options(**sf_options).option("dbtable", "orders_out").mode("append").save()
  • Use dbtable for a whole table or query for a SQL statement. A query is run by Snowflake, so filters and aggregates are computed in Snowflake and only the result reaches Spark. This is called query pushdown.
  • The connector moves data through a temporary stage, so writes use bulk loading rather than row by row inserts.
  • Compute on the Snowflake side uses the virtual warehouse named in the options.
  • A cloud storage location for the temporary stage can be supplied when needed.

Python connector

The Python connector (snowflake-connector-python) is a driver that implements the Python database API.

import snowflake.connector
con = snowflake.connector.connect(account="org-acct", user="svc_etl",
        private_key_file="rsa_key.p8", warehouse="etl_wh", database="raw")
cur = con.cursor()
cur.execute("SELECT * FROM orders WHERE region = %s", ("EU",))
df = cur.fetch_pandas_all()
  • Binding: use bind variables (%s by default, or qmark and numeric styles) rather than string concatenation, which prevents SQL injection.
  • pandas: fetch_pandas_all() and fetch_pandas_batches() need the pandas extra of the package. write_pandas() loads a DataFrame through a stage.
  • Async: execute_async() starts a query without waiting; the query ID (cursor.sfqid) lets the program poll or fetch the results later.
  • Authentication: password, key pair, OAuth and browser based SSO are supported. Service accounts should use key pair or another non interactive method.
  • Snowpark versus connector: the connector sends SQL text and returns result sets. Snowpark builds DataFrame operations that Snowflake executes as SQL on the server.

Native connectors

Native connectors are Snowflake built connectors that bring data from a specific source, such as a database or a SaaS application, into a Snowflake account. They are installed in the account, configured with source connection details and then ingest on a schedule or continuously, so there is no connector code to write.

Snowflake's newer integration service, Openflow, now offers many source specific connectors, including PostgreSQL, MySQL, SQL Server, Oracle, Salesforce and Kafka. For a current choice, check the connector list in the documentation, since the delivery method (Native App or Openflow) can differ by source.

Common mistakes

  • Choosing the Snowpipe path of the Kafka connector and expecting second level latency. File buffering adds delay.
  • Expecting row order from Kafka to be kept in the table.
  • Reading a whole table into Spark and filtering there instead of passing a query that Snowflake filters first.
  • Building SQL strings with user input in the Python connector instead of binding variables.
  • Using the Python connector to run heavy transformations in the client. Push the work to Snowflake with SQL or Snowpark.

What to remember for the exam

  • The Kafka connector writes RECORD_CONTENT and RECORD_METADATA VARIANT columns, uses key pair authentication, and ingests with Snowpipe or Snowpipe Streaming.
  • Kafka classic v3 is planned for deprecation, and v4 is recommended for new work.
  • The connector has no extra fee. Snowpipe or Snowpipe Streaming costs and storage apply.
  • Spark connector: format net.snowflake.spark.snowflake, dbtable or query, query pushdown, temporary stage for transfer.
  • Python connector: connect, cursors, bind variables, fetch_pandas_all, write_pandas, execute_async.
  • Native connectors ingest from a named source with no code, and Openflow is the newer service for the same job.

Check yourself

  • Which two VARIANT columns does the Kafka connector create by default?

    RECORD_CONTENT for the message and RECORD_METADATA for topic, partition, offset and other details.

  • How does the Spark connector make sure filtering happens in Snowflake?

    Pass a query option, so Snowflake runs the SQL and only the result is returned to Spark (query pushdown).

  • How should a Python program pass user supplied values into a query?

    Use bind variables in cursor.execute() instead of building the SQL string.

Topic 11 of 17 in Data Movement

Test yourself on Data Movement

The full Snowflake DEA-C02 question bank has 709 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Snowflake. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.