Ingest and Transform Data · Topic 5 of 6
Streaming Engines, Shortcuts in Real-Time Intelligence and Eventstream
DP-700 streaming: choose Eventstream, Eventhouse or Spark, native tables vs OneLake shortcuts, query acceleration, and Eventstream sources and operators.
- 5 min
- Includes a diagram
- Eventstream captures and routes; Eventhouse stores and queries; Spark streams with code into Delta.
- Shortcuts in a KQL database are queried with
external_table(). - Native tables are fastest but need ingestion; standard shortcuts avoid copies but are slower.
What it is
Real-Time Intelligence is the part of Fabric for data in motion. Eventstream captures, transforms and routes events; an Eventhouse (with one or more KQL databases) stores and queries them; Activator acts on them; and the Real-Time hub lists every stream you can use.
You also decide whether an Eventhouse holds data in native tables or reads Delta data in OneLake through shortcuts, and whether those shortcuts use query acceleration.
How it works
Choose an appropriate streaming engine
| Engine | Choose it when |
|---|---|
| Eventstream | You need no-code capture from many sources, light transformation and routing to several destinations |
| Eventhouse with KQL | You need to store high volumes of events and query them by time in seconds, with update policies and materialized views |
| Spark structured streaming | You need code-first streaming in a notebook or Spark job definition, complex logic or joins with large lakehouse tables, written to Delta |
They combine: Eventstream often feeds an Eventhouse or a lakehouse, and Spark can read the resulting Delta tables. Pick Eventhouse and KQL for log and telemetry analytics, and Spark when the team works in PySpark and the target is a lakehouse.
Native tables or OneLake shortcuts
A KQL database can query Delta tables in OneLake through a OneLake shortcut. It appears as an external table and is queried with external_table().
external_table("SalesHistory")
| where OrderDate > ago(30d)
| summarize Revenue = sum(Amount) by Region| Native table | Standard shortcut | Accelerated shortcut | |
|---|---|---|---|
| Where the data lives | Eventhouse storage | OneLake, as Delta | OneLake, as Delta, indexed and cached by the Eventhouse |
| Ingestion needed | Yes | No | No |
| Query speed | Fastest | Slowest; reads Delta files at query time | Close to native for the cached period |
| Extra cost | Ingestion and storage | None beyond queries | Cache storage and indexing compute |
| Best for | Hot event data queried constantly | Occasional queries over reference or historical data | Frequent queries over OneLake data you do not want to ingest |
Query acceleration for OneLake shortcuts
Query acceleration is turned on per shortcut, when you create it or afterwards. The Eventhouse indexes and caches the Delta data, and you choose how many days of data to keep accelerated.
Choose acceleration when users query the shortcut often and need speed close to a native table without duplicating ingestion. Choose a standard shortcut for rare or ad hoc queries, where paying for a cache is wasteful. If data must be transformed with update policies as it arrives, ingest it into a native table instead.
Process data by using Eventstream
Sources include Azure Event Hubs, Azure IoT Hub, a custom endpoint (apps send events using an Event Hubs, AMQP or Kafka client), change data capture from databases such as Azure SQL Database, Apache Kafka services, Amazon Kinesis, Google Cloud Pub/Sub, sample data and Fabric events.
Operators transform events in flight with no code:
| Operator | Does |
|---|---|
| Filter | Keep events that match a condition |
| Manage fields | Add, remove, rename or change the type of fields |
| Aggregate | Running aggregate such as average or sum over a period |
| Group by | Aggregate over a time window (tumbling, hopping, sliding, session, snapshot) |
| Union, Join | Combine streams |
| Expand | Turn an array into one row per element |
A SQL operator lets you write the transformation as a query instead.
Destinations include an Eventhouse, a lakehouse, Activator, a custom endpoint and a derived stream, which publishes the transformed stream to the Real-Time hub for others to use.
When the destination is an Eventhouse, choose between direct ingestion (events go straight into a table, with mapping done in the Eventhouse) and event processing before ingestion (operators run in the eventstream first).
A worked example
Factory sensors send JSON readings to Azure IoT Hub. Engineers need dashboards of the last 24 hours, an alert when temperature exceeds a limit, and five years of history that already sits as Delta in a lakehouse.
An eventstream reads IoT Hub, filters out test devices and routes readings to an Eventhouse native table and to Activator for the temperature rule. The history is exposed in the KQL database as an accelerated OneLake shortcut covering the most queried recent months, so analysts can join history with live data in KQL without ingesting it again.
How the exam asks about this
- "Route IoT Hub events to an Eventhouse and a lakehouse with no code." Eventstream with two destinations.
- "Query Delta data in OneLake from KQL with near-native performance and no ingestion." OneLake shortcut with query acceleration.
- "Ad hoc KQL queries over a large historical Delta table once a month." A standard OneLake shortcut.
What to remember for the exam
- Eventstream captures and routes; Eventhouse stores and queries; Spark streams with code into Delta.
- Shortcuts in a KQL database are queried with
external_table(). - Native tables are fastest but need ingestion; standard shortcuts avoid copies but are slower.
- Query acceleration indexes and caches shortcut data for a period you choose.
- Eventhouse destinations offer direct ingestion or event processing before ingestion.
Query acceleration is not free. The cache and indexing consume capacity and storage, so choose it only for shortcuts queried often enough to justify it.
Common mistakes
Ingesting large Delta history just to query it occasionally from KQL. A standard shortcut reads it in place without a second copy.
Using Spark structured streaming when the requirement says no code. Eventstream is the no-code option.
Expecting a derived stream to store data. It publishes a transformed stream to the Real-Time hub; storage is in a destination such as an Eventhouse.
Check yourself
How do you query a OneLake shortcut from a KQL database?
With the
external_table()function, using the shortcut's name.When should you prefer an accelerated shortcut to a standard one?
When the shortcut data is queried often and must perform close to a native table, and the cache cost is justified.
What are the two ingestion modes for an Eventhouse destination in Eventstream?
Direct ingestion, and event processing before ingestion.
Topic 5 of 6 in Ingest and Transform Data
Test yourself on Ingest and Transform Data
The full Microsoft DP-700 question bank has 645 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Microsoft. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.