Working with Lakeflow Jobs

What this chapter covers

Working with Lakeflow Jobs is 16% of the Databricks Data Engineer Associate exam. It covers how Databricks orchestrates work: building a job from tasks, wiring their dependencies, controlling what happens on failure or on a condition, and deciding what starts the job.

The chapter has two parts: tasks, dependencies and control flow (retries, If/else branching and For each looping), and job triggers (scheduled, file arrival and table update) with the choice between time based and data driven starts.

Questions are mostly short scenarios in which the stated requirement (latency, cost, governance, data volume) decides the answer. They are single-answer multiple choice, and short SQL or PySpark is shown as text. There are no exhibits.

How the topics fit together

A trigger starts a job run. The job runs its tasks in the order their dependencies set, as a directed acyclic graph (DAG). Control flow decides what happens on the way: a failed task can retry, a condition can pick a branch, and a loop can run one task per input.

• Task questions ask which task type fits and what compute it runs on.
• Dependency questions ask whether a downstream task runs after an upstream failure.
• Control flow questions ask how to retry, branch or loop.
• Trigger questions ask which trigger starts the job at the right moment for the stated data arrival.

How to use these pages

Each topic page has the same shape, so you can skim for what you need.

• What it is: the idea in plain words.
• How it works: the mechanism, with tables and examples.
• What to remember for the exam: the facts to memorize and how questions are phrased.
• Common mistakes: the classic traps.
• Check yourself: three questions to test your understanding.