Data Ingestion and Transformation · Topic 7 of 8
MWAA and Event Driven Pipelines
Use Amazon MWAA, EventBridge, SNS and SQS to orchestrate event driven and scheduled data pipelines, and choose between Airflow and Step Functions on AWS.
- 5 min
- Includes a diagram
- MWAA is managed Apache Airflow. DAGs, plugins and requirements live in S3.
- Choose MWAA when Airflow skills or existing DAGs are stated. Otherwise Step Functions is the AWS native choice.
- Airflow supports sensors, retries, schedules and backfills.
What it is
Some pipelines are driven by a clock, and others by events such as a file arriving. AWS provides managed Apache Airflow for code defined workflows, and a set of messaging and event services to connect the parts of a serverless design.
This page covers Amazon Managed Workflows for Apache Airflow (MWAA), Amazon EventBridge, Amazon SNS (Simple Notification Service) and Amazon SQS (Simple Queue Service).
How it works
Amazon MWAA
MWAA runs Apache Airflow without you managing the servers. You write a workflow as a DAG (directed acyclic graph) in Python, upload it to an S3 bucket, and MWAA reads it from there. Plugins and a requirements.txt file for extra Python packages are also stored in S3.
Airflow tasks use operators and sensors. Operators do work, such as starting a Glue job or an EMR step, and sensors wait for a condition, such as a file in S3. Airflow supports task dependencies, retries, schedules and backfills, which means re running a DAG over past dates.
from airflow import DAG
from airflow.providers.amazon.aws.operators.glue import GlueJobOperator
from datetime import datetime
with DAG("daily_orders", start_date=datetime(2026, 1, 1),
schedule="@daily", catchup=False) as dag:
transform = GlueJobOperator(task_id="transform",
job_name="curate-orders")MWAA sends task and scheduler logs to Amazon CloudWatch Logs. It needs an execution role and a VPC, and it is not serverless in the way Step Functions is, since you choose an environment class.
Amazon EventBridge
EventBridge routes events from AWS services, your applications and SaaS partners to targets. A rule matches events with an event pattern and sends them to targets such as Lambda, Step Functions, SQS, SNS or a Glue workflow. A schedule rule, or EventBridge Scheduler, starts things at fixed times using cron or rate expressions.
Many AWS services publish events to EventBridge, for example a Glue job state change event. A rule on that event can alert a team or start the next stage.
SNS and SQS
SNS is a publish and subscribe service. One message published to a topic is delivered to every subscriber, such as email, Lambda, SQS queues or HTTP endpoints.
SQS is a queue. Producers send messages, and consumers poll and process them, so the queue buffers bursts and decouples the two sides. A message that fails repeatedly can move to a dead letter queue (DLQ) for later inspection.
| Feature | SNS | SQS |
|---|---|---|
| Model | Publish and subscribe, push | Queue, consumers poll |
| Delivery | To all subscribers | To one consumer per message |
| Use | Fan out and notifications | Buffering and decoupling |
The common fan out pattern sends one SNS message to several SQS queues, each served by a different consumer.
Choosing an orchestrator
| Need | Choice |
|---|---|
| Team already writes Airflow DAGs, or needs to move existing Airflow | MWAA |
| Serverless, pay per transition, tight AWS integration | Step Functions |
| Only Glue crawlers and jobs | Glue workflows |
| Loose coupling between services through events | EventBridge with SNS or SQS |
MWAA suits complex dependencies, many external systems and backfills. Step Functions suits workflows that are mostly AWS service calls.
Serverless pipeline design
A serverless data pipeline can chain these pieces. S3 raises an event, EventBridge or SQS carries it, Lambda or Step Functions starts the work, and SNS reports the result. Each piece scales on its own and you pay for use.
Use SQS between stages when downstream work is slow, so spikes do not overload it. Use a DLQ so that one poison message does not block the queue.
Notifications and alerts
SNS is the standard way to alert people about pipeline outcomes. Step Functions and MWAA tasks can publish to SNS on failure. Glue job state changes reach SNS through EventBridge rules, which can filter events first and send only the failures.
What to remember for the exam
- MWAA is managed Apache Airflow. DAGs, plugins and requirements live in S3.
- Choose MWAA when Airflow skills or existing DAGs are stated. Otherwise Step Functions is the AWS native choice.
- Airflow supports sensors, retries, schedules and backfills.
- EventBridge rules match event patterns, and schedules use cron or rate expressions.
- SNS fans out to many subscribers. SQS buffers for one consumer group.
- A dead letter queue captures messages that keep failing.
- SNS plus SQS is the standard fan out pattern.
- Glue job state changes are published as events that EventBridge can route.
If the scenario says "the team already uses Apache Airflow", the answer is MWAA, not a rewrite in Step Functions.
Common mistakes
Treating MWAA as fully serverless. You pick an environment class and it runs continuously, so it has a base cost.
Using SNS where buffering is needed. SNS pushes immediately. Add an SQS queue when consumers need to work at their own pace.
Skipping the DLQ. Without one, a bad message is retried until it expires or blocks progress.
Mixing up SQS queue types. A standard queue gives at least once delivery, while a FIFO (first in, first out) queue keeps order within a message group.
Standard SQS queues can deliver a message more than once, so consumers must be idempotent.
Check yourself
Where does MWAA read DAG files from?
An Amazon S3 bucket.
Which service pushes one message to many subscribers?
Amazon SNS.
What stores messages that fail processing repeatedly?
A dead letter queue.
Topic 7 of 8 in Data Ingestion and Transformation
Test yourself on Data Ingestion and Transformation
The full AWS DEA-C01 question bank has 845 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.