Design data storage solutions · Topic 5 of 5
Data Integration and Analysis
AZ-305 data platform design: choose Data Factory, Synapse, Microsoft Fabric, Databricks, Stream Analytics and Event Hubs for integration and analysis workloads.
- 4 min
- Data Factory orchestrates batch ETL and ELT. Event Hubs ingests streams and Stream Analytics processes them with SQL.
- Fabric is the unified SaaS platform. Synapse is the integrated workspace. Databricks is the Spark lakehouse.
- The self hosted integration runtime reaches on premises and firewalled sources.
What it is
Data integration moves and transforms data from many sources into a form that can be analysed. Data analysis then turns it into insight with warehouses, lakes, Spark and BI. Azure has several overlapping services here, and the exam asks you to pick the one that fits: batch or streaming, code or no code, warehouse or lake, managed or open source.
How it works
Data integration services
| Service | Use |
|---|---|
| Azure Data Factory | Managed ETL and ELT orchestration: pipelines, copy activities, mapping data flows and 100 plus connectors. Includes a self hosted integration runtime for on premises sources |
| Synapse pipelines | The same pipeline engine inside a Synapse workspace |
| Microsoft Fabric Data Factory | Pipelines and dataflows inside the Fabric SaaS platform |
| Azure Event Hubs | Ingest millions of streaming events per second, with Kafka compatibility |
| Azure IoT Hub | Device to cloud ingestion with device management |
| Azure Stream Analytics | SQL like real time processing of streams, no infrastructure |
| Azure Data Box and Storage Mover | Bulk and offline movement (migration chapter) |
Data Factory is the default choice for scheduled batch movement with a visual designer. Event Hubs plus Stream Analytics (or Spark Structured Streaming) is the default for real time. The self hosted integration runtime connects to sources behind a firewall, and managed virtual network integration runtimes use private endpoints for secure access to PaaS sources.
Analytics platforms
| Platform | Strength |
|---|---|
| Microsoft Fabric | Unified SaaS analytics: OneLake, lakehouse, warehouse, data engineering, real time analytics and Power BI in one product with capacity based pricing |
| Azure Synapse Analytics | Integrated workspace: dedicated SQL pools (MPP warehouse), serverless SQL, Spark and pipelines |
| Azure Databricks | Apache Spark based lakehouse. Best for data engineering, machine learning and open source skills (Delta Lake) |
| Azure HDInsight | Managed open source clusters (Hadoop, Spark, Kafka) |
| Azure Data Explorer | Very fast log and time series analytics with KQL |
| Power BI | Reporting and dashboards |
| Azure Analysis Services | Semantic models |
Microsoft's strategic direction is Fabric, so a modern greenfield answer often favours it, while Databricks stays strong where teams already use Spark and Delta Lake. Synapse dedicated SQL pools suit a classic MPP data warehouse. Serverless SQL queries files in the lake on demand, paying per data scanned.
Design patterns
- Lakehouse and medallion architecture: bronze (raw), silver (cleaned) and gold (curated) layers in Data Lake Storage Gen2 or OneLake, using Delta tables.
- Lambda or Kappa: separate or unified batch and streaming paths.
- Operational to analytical: use Synapse Link or Fabric mirroring to analyse Cosmos DB or SQL data without affecting the transactional system.
- Governance: Microsoft Purview catalogs, classifies and traces lineage across sources.
- Security: use managed identities and private endpoints, and keep secrets in Key Vault.
Choosing quickly
| Requirement | Answer |
|---|---|
| Copy nightly data from on premises SQL Server to the lake | Data Factory with a self hosted integration runtime |
| Process IoT telemetry in real time with SQL like queries | Event Hubs or IoT Hub with Stream Analytics |
| Data science team using Spark notebooks and Delta | Azure Databricks (or Fabric data engineering) |
| One SaaS platform from ingestion to Power BI | Microsoft Fabric |
| Classic enterprise data warehouse with MPP | Synapse dedicated SQL pool |
| Analyse application logs and telemetry at scale | Azure Data Explorer |
| Analyse Cosmos DB data without hurting the app | Synapse Link or analytical store |
A worked scenario
A retailer loads sales data nightly from stores, streams clickstream events, and wants one governed analytics platform. The architect uses Data Factory pipelines (with a self hosted integration runtime) for nightly loads into a Fabric lakehouse in medallion layers. Clickstream flows through Event Hubs and Stream Analytics into the lakehouse. Data scientists use Spark notebooks, Power BI serves the business, and Purview records lineage and classifications.
What to remember for the exam
- Data Factory orchestrates batch ETL and ELT. Event Hubs ingests streams and Stream Analytics processes them with SQL.
- Fabric is the unified SaaS platform. Synapse is the integrated workspace. Databricks is the Spark lakehouse.
- The self hosted integration runtime reaches on premises and firewalled sources.
- Synapse Link and Fabric mirroring analyse operational data without impacting it.
- A medallion lakehouse uses bronze, silver and gold layers with Delta tables.
"Real time with SQL like queries and no infrastructure" is Stream Analytics. "Orchestrate scheduled copies" is Data Factory.
How the exam asks about this
- "Move data from an on premises database on a schedule." Data Factory with a self hosted integration runtime.
- "Ingest millions of events per second." Event Hubs.
- "Data engineers want Spark and Delta Lake with notebooks." Azure Databricks.
- "Analyse operational Cosmos DB data without extra load." Synapse Link.
Common mistakes
Using Data Factory for real time stream processing. It is batch oriented.
Choosing HDInsight for new projects. Prefer Fabric, Synapse or Databricks.
Forgetting the integration runtime for private sources.
Querying the operational store for analytics.
Check yourself
Which service orchestrates scheduled data movement?
Azure Data Factory.
Which service ingests high volume streaming events?
Azure Event Hubs.
Which platform is Microsoft's unified SaaS analytics product?
Microsoft Fabric.
Topic 5 of 5 in Design data storage solutions
Test yourself on Design data storage solutions
The full Microsoft AZ-305 question bank has 755 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Microsoft. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.