Databricks Intelligence Platform · Topic 1 of 1

Platform Architecture, Delta Lake, Unity Catalog and Compute

Databricks DEA: control and compute planes, Delta Lake, the Unity Catalog hierarchy, and choosing serverless, jobs, all purpose or SQL warehouse compute.

5 min

In a nutshell

The control plane holds the workspace, scheduler and Unity Catalog services; data is processed in the compute plane and stored in cloud object storage.
A Delta table is Parquet files plus the _delta_log transaction log, giving ACID writes, schema enforcement and time travel.
Unity Catalog names are catalog.schema.object; volumes hold files.

What it is

The Databricks Intelligence Platform is a lakehouse: data sits in open files in cloud object storage, and the platform adds reliable tables, governance and compute on top. Three ideas carry most exam questions: the architecture (control plane and compute plane), Delta Lake (the table format) and Unity Catalog (governance).

The fourth idea is compute. Several compute types exist, and each one fits a different workload on cost, startup time and what it supports.

How it works

Control plane and compute plane

The control plane runs in the Databricks account. It holds the web application, notebooks, the job scheduler and the Unity Catalog services.

The compute plane is where data is processed. Classic compute runs in your own cloud account; serverless compute runs in a compute plane that Databricks manages. Table data stays in cloud object storage either way.

Delta Lake

Delta Lake is the default table format. A Delta table is a set of Parquet data files plus a transaction log in the _delta_log folder. The log is what gives the table its guarantees.

Feature	What it gives you
ACID transactions	Readers never see a half finished write
Schema enforcement	Writes that do not match the table schema are rejected
Time travel	Query an older version with `VERSION AS OF` or `TIMESTAMP AS OF`
Table history	`DESCRIBE HISTORY` lists each operation on the table

SELECT * FROM main.sales.orders VERSION AS OF 12;
DESCRIBE HISTORY main.sales.orders;

Unity Catalog

Unity Catalog governs data and AI assets in one place across workspaces. Objects are addressed with a three level name: catalog.schema.object.

A volume holds non tabular files, such as raw JSON or images, under a path like /Volumes/catalog/schema/volume/. Unity Catalog also records lineage and audit logs, so one place answers "who can read this" and "where did it come from".

Compute types

Compute	Best fit	Key traits
All purpose (classic)	Interactive development in notebooks	You create and share it; it runs until it auto terminates
Jobs compute (classic)	Scheduled production jobs	Created for the run and terminated when it ends; lower rate than all purpose
Serverless (notebooks, jobs, pipelines)	Fast start with no cluster tuning	Databricks manages capacity; starts in seconds; billed for usage
SQL warehouse	SQL queries, dashboards, BI tools	Serverless, pro or classic types; serverless is recommended

All compute is billed in Databricks Units (DBUs) on top of any cloud cost. The exam asks for the cheapest suitable option, not a price: a job on all purpose compute pays the higher interactive rate and leaves a cluster running.

Classic compute has an access mode. Standard (formerly shared) lets several users share the compute with isolation between them. Dedicated (formerly single user) assigns it to one user or group. Instance pools keep idle instances ready to shorten classic startup.

Serverless limitations

Serverless removes cluster management, so some low level controls are gone.

R is not supported, and Scala is not supported in notebooks.
RDD APIs do not work; only Spark Connect APIs do.
Most Spark configuration settings and compute scoped init scripts are not supported.
Libraries are installed at notebook scope or through the environment, not on the compute.

When a workload needs one of these, choose classic jobs or all purpose compute instead.

How the exam asks about this

"A nightly notebook job must run at the lowest cost with no idle cluster." Jobs compute or serverless jobs, not an all purpose cluster.
"Analysts run SQL dashboards and need instant startup." A serverless SQL warehouse.
"A legacy job uses RDD code and a custom Spark config." Classic compute, because serverless does not support either.

What to remember for the exam

The control plane holds the workspace, scheduler and Unity Catalog services; data is processed in the compute plane and stored in cloud object storage.
A Delta table is Parquet files plus the _delta_log transaction log, giving ACID writes, schema enforcement and time travel.
Unity Catalog names are catalog.schema.object; volumes hold files.
All purpose compute is for interactive work, jobs compute for scheduled runs, SQL warehouses for SQL and BI.
Serverless starts in seconds and needs no tuning, but drops R, RDDs, most Spark configs and init scripts.
Access modes are Standard (shared) and Dedicated (single user or group).

Watch out

Running production jobs on an all purpose cluster works, but it is never the cost optimized answer. Jobs compute ends with the run; an all purpose cluster bills at a higher rate until it terminates.

Common mistakes

Thinking Databricks stores your table data in the control plane. Table files live in cloud object storage. The control plane holds the workspace, metadata services and scheduling.

Treating Delta Lake as a separate database. Delta Lake is a file format plus a transaction log. Its features come from the log, not from a server.

Picking serverless for everything. Serverless is the default good answer for speed and simplicity, but a requirement for R, RDD code or custom Spark settings rules it out.

Check yourself

What makes a folder of Parquet files a Delta table?
The _delta_log transaction log, which records every commit and gives the table ACID writes, schema enforcement and time travel.
Which compute fits a scheduled production notebook at the lowest cost?
Jobs compute or serverless jobs. Jobs compute is created for the run, ends with it and bills at a lower rate than all purpose compute.
Why might a job fail to move to serverless compute?
Serverless does not support RDD APIs, R, most Spark configuration settings or compute scoped init scripts, so code that needs them stays on classic compute.

Topic 1 of 1 in Databricks Intelligence Platform

Test yourself on Databricks Intelligence Platform

The full Databricks Data Engineer Associate question bank has 598 questions, with timed practice exams and an explanation for every answer.

Get Full Access Try Sample Exam

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.