Tasks, Dependencies and Control Flow

What it is

A Lakeflow Job is a set of tasks that Databricks runs together. Each task does one unit of work, such as running a notebook or refreshing a pipeline, and Depends on links set the order. The links form a directed acyclic graph (DAG), shown as the task graph in the Jobs UI.

Control flow adds logic to the graph: retries for failures, run if conditions on dependencies, If/else tasks for branching and For each tasks for looping.

How it works

Common task types

Other task types include Python script, Python wheel, JAR, dbt, Run Job (start another job), If/else and For each. A Run Job task lets one job call another, so shared work is defined once.

Dependencies and the DAG

A task with no dependencies starts as soon as the run starts, so independent tasks run in parallel. A task with several upstream tasks waits until they finish, then checks its Run if condition.

With the default, a failed upstream task means the downstream task does not run and the job run fails. All done suits a cleanup or notification task that must run either way; At least one failed suits an alert task.

Passing values between tasks

A notebook task can publish a task value that later tasks read through a dynamic reference. Job parameters are read the same way.

dbutils.jobs.taskValues.set(key="row_count", value=df.count())
# referenced later as {{tasks.load_orders.values.row_count}}
# job parameter as {{job.parameters.env}}

Retries, timeouts and repair

Each task can have a retry policy: how many times to retry after a failure and how long to wait between attempts. By default, most job configurations do not retry, though serverless jobs auto-optimize retries. A task timeout sets the maximum duration; when both are set, the timeout applies to each retry.

Retries fit transient failures, such as a source API that is briefly unavailable. They do not fix a code bug.

Repair run reruns only the failed or canceled tasks of a run and the tasks that depend on them, so successful tasks are not redone. A repaired task starts from the beginning, so a task that wrote part of its output before failing must be safe to rerun.

Branching with If/else

An If/else condition task compares two operands with ==, !=, >, >=, < or <=. The operands are usually a task value or job parameter reference and a literal. == and != compare as strings; the others compare as numbers.

Downstream tasks pick a branch in their Depends on field: check_rows (true) or check_rows (false).

Looping with For each

A For each task runs one nested task once per element of an input array. The inputs are a JSON array, a task value reference or a job parameter. Inside the nested task, {{input}} is the current element and {{input.<key>}} is a field of an object element.

The Concurrency setting controls how many iterations run in parallel; the default is 1. A common use is one ingestion notebook run per source table or region.

How the exam asks about this

• "A cleanup task must run even when the transform task fails." Set its Run if to All done.
• "Process 40 regions with the same notebook, several at a time." Use a For each task with a higher Concurrency.
• "Publish only if the load produced rows; otherwise notify the team." Set a task value and branch with an If/else task.