Data Ingestion and Loading · Topic 1 of 3
Ingestion Patterns and Choosing a Method
Databricks DEA: batch, streaming and incremental ingestion, file upload, Lakeflow Connect standard and managed connectors, and choosing the right method.
- 5 min
- Batch reloads everything, incremental loads only what is new, streaming processes data continuously.
- Auto Loader with an available now trigger is incremental batch: it processes new files, then stops.
- Standard connectors cover cloud storage, SFTP and message buses; managed connectors cover databases and SaaS apps.
What it is
Ingestion is the first step of every pipeline: copying source data into a bronze Delta table that Unity Catalog governs. The exam asks two things about it. Which pattern fits the timing need (batch, streaming or incremental), and which method fits the source.
Lakeflow Connect is the umbrella name for Databricks ingestion. It has managed connectors for databases and SaaS applications, and standard connectors for files and message buses, built on Auto Loader and Structured Streaming.
How it works
Batch, streaming and incremental
| Pattern | What happens | Fits |
|---|---|---|
| Full batch | Reads the whole source on each run and overwrites or reloads | Small reference data, one time loads |
| Incremental batch | Each scheduled run loads only new data since the last run | Daily or hourly file drops |
| Streaming | Runs continuously and processes data as it arrives | Low latency needs, event streams |
Incremental is the default good answer for growing data. Auto Loader and COPY INTO both track what they already loaded, so a rerun does not duplicate rows. Auto Loader can also run as a scheduled batch: a stream with an available now trigger processes all new files and then stops.
Importing local files
For a one off file on a laptop, use the Add data UI to create or modify a table from a file upload. You can also upload files into a Unity Catalog volume and load them from there. This is for small, manual loads, not recurring pipelines.
Standard connectors
Standard connectors cover sources where you write the ingestion logic yourself, with more control and less automation.
| Source | Typical method |
|---|---|
| Cloud object storage (S3, ADLS, GCS) | Auto Loader or COPY INTO |
| SFTP servers | SFTP file ingestion, an extension of Auto Loader |
| Apache Kafka, Amazon Kinesis | Structured Streaming or a Lakeflow pipeline |
Auto Loader runs from Structured Streaming in Python, from a Lakeflow Declarative Pipeline, or from Databricks SQL as a streaming table.
CREATE OR REFRESH STREAMING TABLE main.bronze.orders
AS SELECT * FROM STREAM read_files(
'/Volumes/main/raw/landing/orders/', format => 'json');Managed connectors
Managed connectors are built and maintained by Databricks for sources such as Salesforce, Workday, ServiceNow, SQL Server, MySQL, PostgreSQL, Google Drive and SharePoint. They handle authentication, change data capture, API changes, retries and schema evolution. Their ingestion pipelines run on serverless compute and write destination tables, mostly streaming tables, with no Spark code to maintain.
Partner and other options
Partner tools such as Fivetran are reached through Partner Connect and suit sources that Databricks has no connector for, when the team already uses that tool. A custom notebook with JDBC or a REST client is the fallback when nothing else fits. Lakehouse Federation queries a source in place without copying it, which avoids ingestion but does not scale like a Delta copy.
Choosing a method
Databricks recommends starting at the most managed layer and moving down only when it does not support your source or need.
| Requirement | Points to |
|---|---|
| Millions of files over time, schema changes | Auto Loader |
| Thousands of files, simple SQL, occasional reload of chosen files | COPY INTO |
| Change data capture from a supported database | Managed connector |
| Low latency events from Kafka | Structured Streaming or a Lakeflow pipeline |
| Least code and maintenance, Unity Catalog governed target | Managed connector |
How the exam asks about this
- "Ingest Salesforce objects into Unity Catalog with the least maintenance." A Lakeflow Connect managed connector.
- "New files land hourly in S3; millions are expected over time, and columns get added." Auto Loader, scheduled as incremental batch or run as a stream.
- "An analyst needs to load one CSV from their laptop once." The Add data file upload UI.
What to remember for the exam
- Batch reloads everything, incremental loads only what is new, streaming processes data continuously.
- Auto Loader with an available now trigger is incremental batch: it processes new files, then stops.
- Standard connectors cover cloud storage, SFTP and message buses; managed connectors cover databases and SaaS apps.
- Managed connectors handle CDC, retries and schema evolution, and their ingestion pipelines run on serverless compute.
- Start at the most managed option and move down only if it cannot meet the requirement.
Do not write a custom JDBC notebook for a source that has a managed connector. The exam treats the managed connector as the answer whenever "least maintenance" or "change data capture" is stated.
Common mistakes
Confusing streaming with incremental. Incremental means only new data is processed. A scheduled incremental batch can use streaming APIs and still stop after each run.
Using file upload for recurring loads. The Add data UI is a manual, one time action. Recurring loads need Auto Loader, COPY INTO or a connector on a schedule.
Ignoring governance in the requirement. When the target must be governed by Unity Catalog, every recommended method writes to a Unity Catalog table; a tool that lands data outside it is the wrong choice.
Check yourself
What does starting at the most managed layer mean when choosing ingestion?
Use a Lakeflow Connect managed connector if one supports the source, and move down to Auto Loader, streaming or custom code only when it does not.
Which method fits millions of files arriving in cloud storage over time?
Auto Loader, which scales to millions of files and has stronger schema inference and evolution than COPY INTO.
How can Auto Loader run as a scheduled batch?
Run the stream with an available now trigger in a scheduled job; it processes all new files since the last run and then stops.
Topic 1 of 3 in Data Ingestion and Loading
Test yourself on Data Ingestion and Loading
The full Databricks Data Engineer Associate question bank has 598 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.