Job Triggers and Schedules

What it is

A trigger decides when a Lakeflow Job starts a run. A scheduled trigger is time based: it fires at set times whether or not new data exists. File arrival and table update triggers are data driven: they fire when new files land in storage or when Unity Catalog tables change.

Jobs can also start manually, from a Run Job task in another job, or through the API.

How it works

Scheduled triggers

A schedule is either simple (every N minutes, hours, days or weeks) or advanced, written as a Quartz cron expression with a time zone. A schedule can be paused without deleting it.

schedule:
  quartz_cron_expression: "0 0 6 * * ?"
  timezone_id: "Europe/London"
  pause_status: UNPAUSED

Quartz cron has a seconds field first, so 0 0 6 * * ? means 06:00:00 every day.

A schedule fits when the source lands at a known time or when the business needs results by a fixed deadline. If the data arrives late, the run still starts and may process nothing or partial data.

File arrival triggers

A file arrival trigger watches the root or a subpath of a Unity Catalog volume or external location on S3, Azure storage or Google Cloud Storage, including subdirectories. It checks for new files about every minute, on a best effort basis.

Requirements: Unity Catalog in the workspace, READ permission on the storage location and CAN MANAGE on the job. Databricks recommends file events on the external location; without them, the monitored location is limited to 10,000 files.

The trigger only starts the job. The job still needs ingestion logic, typically Auto Loader or COPY INTO, to load each new file once.

trigger:
  file_arrival:
    url: /Volumes/main/raw/landing/
    wait_after_last_change_seconds: 120

Table update triggers

A table update trigger watches up to 25 Unity Catalog tables, including managed and external Delta tables, materialized views, streaming tables and supported views. It fires on updates, merges and deletes.

With several tables, choose Any table is updated or All tables are updated. The same two advanced settings, minimum time between triggers and wait after last change, apply.

trigger:
  table_update:
    table_names: [main.silver.orders, main.silver.customers]
    condition: ALL_UPDATED

The job's Run as principal needs SELECT on each monitored table, plus USE CATALOG and USE SCHEMA. Without them, the trigger cannot evaluate and the job does not run.

Time based or data driven

Data driven triggers avoid two costs of polling on a schedule: empty runs when nothing arrived, and delay when data arrives just after a run. A schedule is simpler when arrival is predictable or the deadline matters more than freshness.

When both jobs are yours, a dependency inside the DAG or a Run Job task is more direct than a trigger. Table update triggers suit dependencies across teams or pipelines you do not control.

How the exam asks about this

• "Vendors upload CSV files to a volume at random times; process each upload within minutes." Use a file arrival trigger.
• "Rebuild the gold job only after both silver tables owned by another team have been refreshed." Use a table update trigger with All tables are updated.
• "Finance needs the summary every weekday at 06:00 New York time." Use a scheduled trigger with a Quartz cron expression and the America/New_York time zone.