Batch Ingestion and Migration

What it is

Batch ingestion moves data in chunks, on a schedule or when a file arrives. The source can be a relational database, a SaaS (software as a service) application, files on premises or another AWS account.

The exam tests two skills here. One is matching the source to the right transfer service. The other is designing loads that are safe to run twice, so a retry never creates duplicate rows.

How it works

Matching the source to the service

Pulling from a REST API

A REST (representational state transfer) API that AppFlow does not cover can be consumed with a scheduled Lambda function or a Glue Python shell job. The code calls the endpoint in a loop, follows the pagination token or cursor in each response until none is returned, and writes the raw JSON to S3 unchanged.

• Keep the API key or token in Secrets Manager, never in code.
• On an HTTP 429 (too many requests) or a 5xx error, retry with exponential backoff, and honor a wait hint if the API sends one.
• Store the last cursor or timestamp (for example in DynamoDB or Parameter Store) so a re run resumes cleanly, and write to a dated S3 prefix so a retry overwrites instead of duplicating.
• Lambda stops at 15 minutes, so a long pull belongs in Glue.

Database Migration Service

DMS reads from a source endpoint and writes to a target endpoint. A replication instance runs the task, or you can use DMS Serverless, which scales capacity for you. A task can do a full load, replicate ongoing changes with CDC, or do a full load followed by CDC.

DMS moves data but does not convert a schema between different engines. For a heterogeneous move, such as Oracle to PostgreSQL, use the AWS Schema Conversion Tool or DMS Schema Conversion first. DMS can write to S3, and CDC output there lands as files that carry an operation indicator for inserts, updates and deletes.

Allowlisting source IP addresses

Some sources accept connections only from known IP addresses. AWS jobs do not have a fixed address by default, so you give them one and tell the source about it.

A Lambda function that is not attached to a VPC has no stable outbound address, so it cannot be allowlisted.

Triggers and scheduling

A batch job needs something to start it. Common triggers are an Amazon EventBridge schedule, an S3 event notification, an AWS Glue trigger, an AWS Step Functions state machine or an Amazon Managed Workflows for Apache Airflow (MWAA) DAG (directed acyclic graph).

S3 event notifications can invoke Lambda, send to SQS or SNS, or go to EventBridge. EventBridge supports filtering on object key prefixes and many other fields.

Idempotency and replayability

An idempotent load gives the same result when run once or many times. Replayable means you can re run a past period because the raw input still exists. Keep an untouched copy of source data in a raw S3 layer, and write results so a re run replaces a partition instead of appending to it.

Ways to get idempotency include overwriting a partition, using a merge or upsert on a business key, and using AWS Glue job bookmarks so a job skips data it already processed. In a streaming path, a deduplication key serves the same purpose.

Throttling and limits

Sources and targets push back when you load too fast. Use SQS as a buffer in front of Lambda, set batch sizes, add retries with exponential backoff and cap concurrency. Lambda has reserved concurrency for this, which also protects a database from too many connections.

For Redshift, load with the COPY command from several files in S3 so the nodes read in parallel. Many small single row INSERT statements are slow in comparison.

COPY sales FROM 's3://my-bucket/raw/sales/2026/10/'
IAM_ROLE 'arn:aws:iam::123456789012:role/RedshiftCopyRole'
FORMAT AS PARQUET;

Fan in and fan out

Fan in means many sources feeding one target, such as several Kinesis streams landing in one data lake. Fan out means one event feeding several targets. Amazon SNS with SQS queues is the classic fan out pattern, where each queue serves a different consumer.