Declarative Automation Bundles and the Databricks CLI

What it is

Declarative Automation Bundles (formerly Databricks Asset Bundles) describe a project as code: its source files plus the jobs, pipelines and other workspace resources that run them. The description lives in databricks.yml at the root of the repository.

The same bundle deploys to several targets, such as dev, test and prod. Variables and target overrides change what differs between environments, so the code itself is never copied or edited per environment.

How it works

The configuration file

A bundle has a name, resource definitions and targets. Resources include Lakeflow Jobs (jobs) and Lakeflow Declarative Pipelines (pipelines).

bundle:
  name: sales_etl
resources:
  jobs:
    daily_sales:
      tasks:
        - task_key: load
          notebook_task: { notebook_path: ./src/load.ipynb }

Deploying uploads the files and creates or updates the defined jobs and pipelines in the target workspace. A later deploy updates the same resources instead of creating duplicates.

Variables

Declare custom variables under variables and reference them with ${var.<name>}. A variable without a default must be given a value at deploy time.

variables:
  catalog:
    description: Target catalog
    default: dev_catalog
targets:
  prod:
    variables:
      catalog: prod_catalog

Built-in substitutions also exist, such as ${bundle.target} and ${workspace.current_user.userName}. A lookup variable resolves an object's ID from its name, for example a cluster or SQL warehouse, so IDs are never hard coded.

Variables resolve at deployment time. To pass a value to each job run, use job parameters instead.

Targets and deployment modes

Each target sets a workspace and can override variables and resource settings. One target is usually marked default: true, used when no target is given.

Development mode lets several developers deploy the same bundle to one workspace without overwriting each other or starting scheduled runs.

CLI commands

The Databricks CLI runs every bundle step, locally or in a CI/CD pipeline.

databricks bundle validate -t prod
databricks bundle deploy -t prod
databricks bundle run -t prod daily_sales
databricks bundle destroy -t dev

Promotion in CI/CD

A pull request runs validate and deploys to dev or test. Merging to main deploys the same commit to prod.

The CI system authenticates as a service principal, for example with DATABRICKS_HOST, DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET, not a person's credentials. Production then never depends on one user's account.

How the exam asks about this

• "Prod writes to a different catalog than dev, with the same code." A variable referenced as ${var.catalog} with a target override for prod.
• "Check the bundle for errors in the pipeline before anything is deployed." Run databricks bundle validate.
• "Two developers deploy the same bundle to one dev workspace without clashing." Use a target with mode: development, which prefixes names per user and pauses schedules.