Event Driven Patterns, Streaming, and Lambda Triggers

What it is

In an event driven design, a producer announces that something happened and consumers react. The producer does not know who listens. S3 Event Notifications, EventBridge, SNS and SQS carry these events.

Streaming data is a continuous flow of records, such as clicks or sensor readings. Amazon Kinesis Data Streams and Amazon Data Firehose ingest it, and consumers read it in order.

Lambda is the usual consumer. Which side starts the invocation depends on the source.

How it works

The event sources

The destination must allow the source to call it: a Lambda resource based policy for S3, SNS or EventBridge, or a queue policy for SQS.

An EventBridge rule that reacts to a new S3 object uses a pattern like this one:

{
  "source": ["aws.s3"],
  "detail-type": ["Object Created"],
  "detail": { "bucket": { "name": ["uploads"] } }
}

EventBridge retries a failed target delivery for 24 hours and up to 185 times by default. A dead letter queue on the target keeps undelivered events.

Three ways Lambda is invoked

From code, the Invoke call with InvocationType="Event" starts an asynchronous run, and RequestResponse waits for the result.

An event source mapping is a Lambda resource that reads batches and calls your function synchronously. The function's execution role needs read permission on the source, for example sqs:ReceiveMessage and sqs:DeleteMessage. Managed policies such as AWSLambdaSQSQueueExecutionRole cover these.

Useful mapping settings follow.

• Batch size and batching window control records per invocation.
• Filter criteria drop unwanted events before they invoke the function.
• Partial batch response (ReportBatchItemFailures) lets the function return only the failed item IDs, so the rest are not retried.
• For SQS, set the queue visibility timeout to at least six times the function timeout, and put the dead letter queue on the source queue.

Kinesis Data Streams

A stream is split into shards. The partition key you send picks the shard, and records with the same key keep their order.

• In provisioned mode each shard accepts up to 1 MB per second or 1,000 records per second of writes, and serves 2 MB per second of reads. On demand mode scales shards for you.
• Records are kept for 24 hours by default, and retention can be extended up to 365 days.
• A low cardinality partition key creates a hot shard and ProvisionedThroughputExceededException. Choose a key with many distinct values.
• Consumers share a shard's read throughput. Enhanced fan out gives each registered consumer dedicated throughput.

A Lambda mapping on a stream retries a failing batch until it succeeds or the records expire, and that blocks the shard. Limit the damage with MaximumRetryAttempts, MaximumRecordAgeInSeconds, BisectBatchOnFunctionError (split the batch to isolate the bad record) and an on failure destination.

Amazon Data Firehose is the managed delivery path. It buffers records and writes them to S3, OpenSearch Service and other targets, with optional Lambda transformation. There are no shards to manage, and no replay.

How the exam asks about this

• "Process each image uploaded to S3 with the least operational overhead." -> S3 event notification to a Lambda function, with a prefix filter.
• "Ordered clickstream records must be read by several independent applications and replayed." -> Kinesis Data Streams, not SQS.
• "One bad record keeps a Kinesis consumer stuck." -> Set maximum retry attempts or record age, bisect on error, and an on failure destination.