Automating Data Processing

What it is

Automation removes manual steps from data processing. A pipeline should start on a schedule or on an event, retry on its own after a transient failure, and tell people only when it needs them.

This page looks at the day to day tooling: event source mappings, schedules, service APIs and the settings that make runs repeatable.

How it works

Lambda event source mappings

An event source mapping lets Lambda poll a source and invoke your function with batches. It supports Kinesis Data Streams, DynamoDB Streams, SQS and Amazon MSK among others. The mapping controls the batch size, a batching window and what to do with failed batches.

For SQS, Lambda scales with queue depth, and a dead letter queue holds messages that fail too often. Use a partial batch response so that one bad record does not force the whole batch to retry.

Schedules and events

Amazon EventBridge rules and EventBridge Scheduler run jobs at fixed times or when an event matches a pattern. An S3 event can start a Lambda function when an object arrives, either directly or through EventBridge or SQS.

aws events put-rule --name nightly-etl --schedule-expression "cron(0 2 * * ? *)"
aws events put-targets --rule nightly-etl \
  --targets "Id"="1","Arn"="arn:aws:lambda:us-east-1:123456789012:function:start-etl"

EventBridge cron expressions use six fields and run in UTC for rules. Remember that Lambda also needs a resource policy that lets EventBridge invoke it, which the console adds for you.

Calling services from automation

Most data services have APIs that a function or workflow can call.

The Redshift Data API is asynchronous. A function submits a statement, receives an ID and checks the result later, so it does not hold a database connection open. Redshift also supports scheduled queries from the query editor, and stored procedures run repeatable SQL logic inside the warehouse.

Consuming and maintaining data APIs

A client or workflow that calls a data API through API Gateway authenticates with IAM, a Cognito token or an API key tied to a usage plan. It should retry throttled calls (HTTP 429) with exponential backoff.

To release changes safely, publish each version to a stage, such as dev and prod, or put a version in the path, such as /v1. Keep the old version running until clients have moved.

Monitor the API with CloudWatch metrics, including request count, 4XX and 5XX errors, latency and cache hits and misses. Turn on access logging for the stage to record who called what. Alarm on error rate and latency, and use CloudTrail for changes to the API itself.

Glue and Athena automation

Glue triggers can start jobs on a schedule, on demand, or when other jobs and crawlers succeed. Glue jobs take parameters, so one script can process a different date each run. Athena queries can be scheduled through EventBridge and Step Functions, and CTAS or INSERT INTO statements write results back to S3.

Amazon MWAA covers larger, dependency heavy schedules with sensors and backfills, while Step Functions covers mixed service workflows. The earlier chapter compares them.

Making automation safe

Automated runs are only helpful if they are repeatable and observable.

• Idempotent writes: a re run overwrites a partition or merges on a key.
• Retries with backoff: handle throttling and short outages.
• Concurrency limits: protect databases and APIs from bursts.
• Parameters, not hardcoded values: the same code works in test and production.
• Alerts on failure: SNS or EventBridge notify the team.
• Least privilege roles: each function or job gets only the permissions it needs.

Batch API calls and bulk work

Some APIs accept many items per call, such as PutRecords for Kinesis, BatchWriteItem for DynamoDB and SendMessageBatch for SQS. Batching reduces call count and cost, but you must check the response for failed items, because a batch call can succeed overall while some records fail.