Incident and Event Response · Topic 2 of 5
Queues, Topics and Lambda Processing
Process and notify on events with SNS fan out, SQS queues, dead letter queues, Lambda event source mappings, Kinesis, retries and idempotent handlers.
- 5 min
- Includes a diagram
- SNS fans out; SQS buffers; Kinesis streams in order and can replay.
- SNS plus SQS gives independent, durable consumers.
- A DLQ catches messages that fail repeatedly; alarm on its depth.
What it is
Once an event is routed, something has to process it reliably and tell people when it matters. AWS provides messaging services for this: Amazon SNS for push notification and fan out, Amazon SQS for queues, Kinesis for streams, and AWS Lambda to run code on each message.
The design questions are always the same. What happens when processing fails, what happens when messages arrive faster than you can handle them, and what happens when the same message arrives twice.
How it works
Amazon SNS
SNS publishes a message to a topic, and every subscription receives a copy. Subscribers can be SQS queues, Lambda functions, HTTP endpoints, email, SMS, mobile push and Firehose delivery streams. A subscription filter policy lets each subscriber receive only the messages it wants.
{
"eventType": ["order_placed"],
"amount": [{"numeric": [">", 100]}]
}FIFO topics keep order and deduplicate messages. A subscription can also have a dead letter queue for messages that SNS could not deliver.
Amazon SQS
SQS stores messages until a consumer processes and deletes them.
| Feature | Behavior |
|---|---|
| Standard queue | Nearly unlimited throughput, at least once delivery, order not guaranteed |
| FIFO queue | Ordered within a message group, deduplication, lower throughput |
| Visibility timeout | A received message is hidden from other consumers for this period, then returns if not deleted |
| Long polling | Waits for messages, reducing empty responses and cost |
| Dead letter queue | Receives messages that exceed a maxReceiveCount of failed attempts |
A dead letter queue (DLQ) isolates poison messages, which are messages that always fail, so they do not block or endlessly retry. Set an alarm on the DLQ depth and redrive messages after the bug is fixed.
The SNS plus SQS fan out pattern gives each downstream system its own queue, so a slow or failing consumer never affects the others.
Lambda event source mappings
For queues and streams, Lambda polls the source for you through an event source mapping and invokes your function with a batch.
| Source | Failure behavior |
|---|---|
| SQS | Failed batch returns to the queue after the visibility timeout, then to the DLQ |
| Kinesis and DynamoDB Streams | Lambda retries the batch and blocks that shard until it succeeds or the records expire |
For SQS, set the queue visibility timeout well above the function timeout; AWS suggests about six times the function timeout. Use partial batch responses, so only failed messages return to the queue.
{"batchItemFailures": [{"itemIdentifier": "message-id-that-failed"}]}Enable the ReportBatchItemFailures response type on the mapping, and return the list above from the function.
For Kinesis and DynamoDB streams, order matters inside a shard, so one bad record can stall the shard. Set a maximum retry count, a maximum record age, bisect on error (split the batch to find the bad record) and an on failure destination to capture the rest.
Asynchronous Lambda invocation
When a service such as S3, SNS or EventBridge invokes a function asynchronously, Lambda queues the event and retries on error, by default two more times. Configure the maximum retries, the maximum event age and an on failure destination (SQS, SNS, Lambda or EventBridge) or a DLQ. An on success destination is available as well.
A synchronous invocation, for example from API Gateway, does not retry by itself, and the caller receives the error.
Kinesis Data Streams
Kinesis keeps ordered records in shards for a retention period, and several consumers can read the same data independently. Choose it over SQS when you need ordering by key, replay of old records or many consumers reading the same data.
Notification and incident tooling
- SNS email or SMS for simple alerts.
- Amazon Q Developer in chat applications (formerly AWS Chatbot) posts alarms and events to Slack or Microsoft Teams, and can run limited commands.
- AWS Systems Manager Incident Manager manages response plans, contacts, escalation and runbooks for real incidents.
- AWS Health events through EventBridge announce service issues that affect your resources.
Making handlers safe
Because delivery is at least once, an idempotent handler gives the same result however many times it runs. A common technique is to store an idempotency key, such as the message ID, in DynamoDB with a conditional write, and skip work already done.
What to remember for the exam
- SNS fans out; SQS buffers; Kinesis streams in order and can replay.
- SNS plus SQS gives independent, durable consumers.
- A DLQ catches messages that fail repeatedly; alarm on its depth.
- Visibility timeout must exceed the function processing time.
- Partial batch responses avoid reprocessing the whole SQS batch.
- A failing record in a Kinesis shard blocks that shard, unless you limit retries or age.
- Lambda retries asynchronous invocations twice by default.
- On failure destinations or DLQs capture events that never succeed.
- FIFO queues preserve order within a message group.
- Handlers must be idempotent because delivery is at least once.
When one failing consumer must not slow others, put a separate SQS queue behind an SNS topic for each consumer.
Common mistakes
A visibility timeout shorter than processing time. The message reappears while still being processed and is handled twice.
No DLQ. A poison message retries forever or is lost.
Using SQS where order by key matters. Standard queues give no ordering guarantee.
Treating Kinesis like SQS. Records are not deleted after reading, and a stuck record blocks its shard.
A redrive from a DLQ puts failed messages straight back into the main flow, so fix the cause first or they will fail again.
Check yourself
Which setting prevents a message from being processed twice while a consumer works on it?
The visibility timeout, which hides the message from other consumers for the set period.
How does Lambda avoid reprocessing a whole SQS batch for one bad message?
It returns partial batch responses listing only the failed message IDs.
What happens to a Kinesis shard when a record keeps failing?
Lambda keeps retrying that batch and blocks the shard until it succeeds or the record expires, unless limits are set.
Topic 2 of 5 in Incident and Event Response
Test yourself on Incident and Event Response
The full AWS DOP-C02 question bank has 844 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.