Ways to Bring Data into Databricks

What it is

Data reaches Databricks in several ways, each suited to a different source. Files can be loaded from cloud storage such as Amazon S3, picked up continuously by Auto Loader, pulled from SaaS applications and databases by Lakeflow Connect managed connectors, pushed in by code through APIs, or uploaded by hand in the workspace UI.

Another organization's data need not be copied: Delta Sharing gives read access to live tables across platforms, and Databricks Marketplace lists shared data products.

How it works

Choosing an approach

Files in S3

Unity Catalog reaches S3 through an external location, which pairs a path with a storage credential. With read access, you can query the files or create a table from them.

CREATE TABLE main.sales.raw_orders AS
SELECT * FROM read_files(
  's3://acme-landing/orders/',
  format => 'csv', header => true);

COPY INTO is idempotent, so a rerun loads only new files. Auto Loader is a streaming source (cloudFiles) for files that keep arriving; it tracks processed files in a checkpoint and scales to millions of files.

Lakeflow Connect managed connectors

Managed connectors are point and click ingestion for SaaS applications such as Salesforce, Workday and Jira, and for databases such as SQL Server, MySQL, PostgreSQL and Oracle. You choose the source, objects and destination schema, and Databricks runs an ingestion pipeline that loads incrementally into streaming tables.

There is no connector code to maintain; without a managed connector, the fallback is custom code or a partner tool.

API driven intake

A notebook or job can call a source's REST API and write the JSON response to a table, and Zerobus Ingest lets an application push records directly into Unity Catalog Delta tables through its API.

It suits event data from your own applications and sources with no managed connector.

Delta Sharing

Delta Sharing is an open protocol for sharing live data without replication; current docs also call the platform OpenSharing. A provider creates a share (a read only collection of tables, views and other assets) and a recipient, then grants the share to the recipient.

Nothing is copied, so the recipient sees updates in near real time, read only.

Marketplace

Databricks Marketplace lists datasets, AI models, notebooks and other data products from providers. Some listings are available instantly after you accept the terms; others need the provider's approval. Delivery runs on Delta Sharing, so a Unity Catalog workspace receives the product as a read only catalog.

Uploading a file in the workspace UI

Click New, then Add or upload data, and choose Create or modify a table. You can upload CSV, TSV, JSON, Avro, Parquet or text files.

The preview lets you set the table name and location and adjust column names and types. Creating the table needs CREATE TABLE on the target schema (plus USE CATALOG and USE SCHEMA) and running compute, such as a SQL warehouse.

To keep a file as a file, choose Upload files to a volume, which needs WRITE VOLUME and allows up to 5 GB per file.

How the exam asks about this

• "An analyst has a 50 MB CSV on a laptop and needs to query it today." Upload it in the UI with Create or modify a table, which creates a managed Delta table.
• "A partner on another platform needs read access to a live table without receiving copies." Delta Sharing with open sharing.
• "Load Salesforce objects into Unity Catalog incrementally with no custom code." A Lakeflow Connect managed connector.