Architecture Design and Implementation Built for the Cloud · Topic 3 of 11
Serverless Workflows, Pipelines and Legacy Integration
Pick Step Functions or durable Lambda functions for long workflows, build Kinesis, Glue and Athena pipelines, and modernize legacy apps with Amazon MQ.
- 5 min
- Standard workflows run up to one year; Express workflows run up to five minutes at high volume.
- Durable Lambda functions checkpoint steps and replay to recover, and waits suspend without compute charges.
- Use the saga pattern with compensating steps when a distributed transaction must roll back.
What it is
Serverless building blocks let you process data and run business processes without managing servers. The design choices are how work moves between steps, where progress is saved, and how a failure in step five avoids repeating steps one to four.
The same tools connect old systems to new ones. A legacy app that speaks a message broker protocol or an old payload format can join a modern architecture through a translation layer.
How it works
Serverless data pipelines
- Kinesis Data Streams captures ordered records in real time. Lambda reads it through an event source mapping in batches, and the batch size and batch window trade latency for efficiency.
- Amazon Data Firehose delivers a stream to S3, Redshift or OpenSearch with buffering, and can convert records to columnar formats such as Parquet.
- AWS Glue runs serverless Spark or Python ETL jobs, and crawlers populate the Glue Data Catalog with table schemas.
- Amazon Athena runs SQL directly on S3 using the catalog. You pay by data scanned, so Parquet, compression and partitioning cut both cost and query time.
Use Lambda for light, per-record transforms and Glue for heavy joins and large batch transformations. Orchestrate the batch steps with Step Functions or Glue workflows.
Durable multi-step workflows
| Option | Defined as | Duration | Execution semantics | Fits |
|---|---|---|---|---|
| Step Functions Standard | State machine (Amazon States Language, visual designer) | Up to one year | Exactly once workflow execution, full history | Long business processes, human approval, auditable flows |
| Step Functions Express | State machine | Up to five minutes | At least once (asynchronous) or at most once (synchronous) | High volume, short event processing |
| Lambda durable functions | Ordinary code with the durable execution SDK | Up to one year | Checkpoint and replay of completed steps | Workflows tightly coupled to application code in Lambda |
Lambda durable functions wrap work in steps that are checkpointed, and waits that suspend the function without compute charges. After an interruption the function replays from the start and skips completed steps by using their stored results.
Step Functions is the stronger choice when you need visual design, direct integrations with many AWS services, or orchestration across teams. Durable functions suit developers who want the workflow to live inside the Lambda code.
Durable patterns that the exam names
- Callback with task token: Step Functions pauses until an external system or a human returns the token, which suits approvals.
- Saga: each step has a compensating action. If a later step fails, earlier steps are undone in reverse order.
- Retry and catch: per-state retry with backoff, then a catch path to a fallback or a dead-letter destination.
- Distributed Map: runs many child workflows in parallel over a large dataset, such as millions of objects in S3.
Modernizing legacy systems
| Legacy situation | Integration pattern |
|---|---|
| App uses JMS, AMQP, MQTT, STOMP or OpenWire with Apache ActiveMQ or RabbitMQ | Amazon MQ, so the app moves without rewriting messaging code |
| Old SOAP or XML backend behind a modern REST client | API Gateway REST API with mapping templates or a Lambda function that transforms the payload |
| Monolith being replaced piece by piece | Strangler fig: API Gateway routes some paths to new services and the rest to the monolith |
| Point to point messages that need filtering and enrichment | EventBridge Pipes, with filtering, enrichment and input transformation between a source and a target |
Choose Amazon MQ when protocol compatibility matters. For new cloud-native messaging with no protocol constraint, SQS and SNS scale further with less to manage.
How the exam asks about this
- "A loan approval waits days for a human decision and every step must be auditable" -> Step Functions Standard with a callback task token.
- "An on-premises app uses JMS with ActiveMQ and must move to AWS with minimal code change" -> Amazon MQ for ActiveMQ.
- "Clickstream data must be queryable with SQL within minutes at the lowest operational overhead" -> Kinesis to Firehose to partitioned Parquet in S3, cataloged by Glue and queried with Athena.
What to remember for the exam
- Standard workflows run up to one year; Express workflows run up to five minutes at high volume.
- Durable Lambda functions checkpoint steps and replay to recover, and waits suspend without compute charges.
- Use the saga pattern with compensating steps when a distributed transaction must roll back.
- Athena charges by data scanned, so columnar formats and partitions are the main cost levers.
- Amazon MQ is the lift-and-shift path for standard broker protocols. SQS and SNS are the cloud-native path.
- API Gateway mapping templates or Lambda translate legacy payloads without changing the backend.
Do not chain Lambda functions by having each one call the next synchronously. Errors, timeouts and retries become hard to track and you pay for waiting; use Step Functions or durable functions to hold the state.
Common mistakes
Picking Express for a long approval process. Express workflows stop at five minutes, so any human or multi-day wait needs Standard or a durable function.
Rewriting a broker-based app to SQS during migration. When the requirement is minimal code change, Amazon MQ keeps the existing protocol and client libraries.
Querying raw JSON in S3 with Athena at scale. Unpartitioned JSON makes every query scan everything. Convert to Parquet and partition by the common filter, such as date.
Check yourself
How does a Lambda durable function avoid repeating finished work after an interruption?
It replays from the start and skips completed steps by returning their checkpointed results instead of running them again.
Which Step Functions pattern lets a workflow wait for a human decision?
A task that waits for a callback task token, which the approver's system returns to resume the workflow.
When is Amazon MQ a better choice than SQS?
When an existing application uses a standard broker protocol such as JMS or AMQP and must migrate without rewriting its messaging code.
Topic 3 of 11 in Architecture Design and Implementation Built for the Cloud
Test yourself on Architecture Design and Implementation Built for the Cloud
The full AWS SAP-C03 question bank has 850 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.