Data Ingestion and Loading

What this chapter covers

Data Ingestion and Loading is 21% of the Databricks Data Engineer Associate exam, the second largest share after Data Transformation and Modeling. It covers how data gets from its source into Unity Catalog governed Delta tables.

The chapter has three parts: ingestion patterns and choosing a method, loading files from cloud storage with COPY INTO and Auto Loader, and ingesting from enterprise sources with Lakeflow Connect, JDBC and REST clients, including semi structured JSON.

Questions are mostly short scenarios in which the stated requirement (latency, cost, governance, data volume) decides the answer. They are single-answer multiple choice, and short SQL or PySpark is shown as text. There are no exhibits.

How the topics fit together

Every ingestion question starts with the source and the requirement. Files in cloud storage point to COPY INTO or Auto Loader. Databases and SaaS applications point to Lakeflow Connect managed connectors, and anything without a connector falls back to code in a notebook.

• Pattern questions ask whether a load is batch, streaming or incremental and which method fits.
• File questions ask about COPY INTO syntax, Auto Loader options and schema evolution.
• Connector questions ask how Lakeflow Connect pipelines are set up and what they create.
• Custom source questions ask how JDBC or REST code in a notebook lands data and how it is scheduled.

How to use these pages

Each topic page has the same shape, so you can skim for what you need.

• What it is: the idea in plain words.
• How it works: the mechanism, with tables and examples.
• What to remember for the exam: the facts to memorize and how questions are phrased.
• Common mistakes: the classic traps.
• Check yourself: three questions to test your understanding.