Design High Performing Architectures · Topic 6 of 7
Data Ingestion and Transfer
Choose how to bring data into AWS: Kinesis Data Streams, Firehose, MSK, DataSync, Snow Family, Transfer Family, Storage Gateway and migration tools.
- 5 min
- Kinesis Data Streams: shards, ordered per partition key, replay within retention, custom consumers.
- Firehose: fully managed, near real time delivery to S3, Redshift, OpenSearch Service and similar targets.
- MSK is managed Kafka, for teams that need Kafka compatibility.
What it is
Before data can be stored or analyzed, it has to arrive. Ingestion covers two broad patterns: streaming data that flows continuously from many producers, and bulk or batch transfer of existing data sets into AWS.
The exam expects you to match the pattern, the volume, the network available and the destination to the right service.
How it works
Streaming ingestion
| Service | What it does | Pick it when |
|---|---|---|
| Kinesis Data Streams | Durable stream of ordered records split into shards, read by your own consumers | You need custom processing, multiple consumers or replay |
| Amazon Data Firehose | Fully managed delivery of streaming data to S3, Redshift, OpenSearch Service and other destinations | You just want to load streaming data with no consumer code, in near real time |
| Amazon MSK | Managed Apache Kafka | You already use Kafka or need Kafka APIs |
| AWS IoT Core | Connects and manages IoT devices over protocols such as MQTT | Devices send telemetry, and rules route it onward |
Kinesis Data Streams capacity comes from shards, and each shard has fixed read and write throughput. You can choose provisioned shards or on demand mode, which scales for you. Records with the same partition key land on the same shard and stay in order, and data is kept for a retention period so consumers can replay it.
Firehose buffers data by size or time and then writes it to the destination. It can convert records to formats like Parquet and run a Lambda function to transform records on the way. Choose it over Data Streams when the requirement is "load into S3 or Redshift with the least management".
Bulk and hybrid transfer
| Service | Use it for |
|---|---|
| AWS DataSync | Online, scheduled transfer of files from NFS, SMB or other storage into S3, EFS or FSx, using an agent, with integrity checks |
| AWS Snow Family (for example Snowball Edge) | Offline transfer of very large data sets, or compute at remote sites, when the network is too slow or unreliable |
| AWS Transfer Family | Managed SFTP, FTPS and FTP endpoints in front of S3 or EFS, so partners keep their existing file transfer tools |
| AWS Storage Gateway | Hybrid access: file, volume and tape gateways let on premises systems use AWS storage with local caching |
| AWS Database Migration Service | Move databases, with ongoing replication to keep downtime small |
| Amazon AppFlow | Move data between software as a service applications and AWS |
Estimate the transfer time first. Terabytes over a fast link may be fine for DataSync.
Petabytes, or a slow link, make a physical Snow device quicker. You fill the device, ship it back, and AWS loads the data into S3.
Ingestion design habits
Land raw data in S3 first, as it is cheap, durable and works with most analytics services. Use partitioned prefixes (for example by date) so later queries read less data. Add a queue or stream between producers and consumers so a slow consumer does not block the producers.
Encrypt data in transit and at rest, and give producers write only access to the ingestion target. Monitor with CloudWatch metrics such as iterator age for Kinesis consumers to see if they are falling behind.
Choosing quickly
- Many producers, custom real time processing, replay: Kinesis Data Streams.
- Load streaming data into S3 or Redshift with no code: Firehose.
- Existing Kafka application: MSK.
- Move files from on premises storage on a schedule: DataSync.
- Huge data set, poor network: Snowball Edge.
- Partners using SFTP: Transfer Family.
- On premises application needs cloud backed storage: Storage Gateway.
When the question includes the words "near real time", "no administration" and "load into S3", Firehose is almost always the answer. Words like "multiple consumers", "replay" or "custom processing" point to Kinesis Data Streams.
What to remember for the exam
- Kinesis Data Streams: shards, ordered per partition key, replay within retention, custom consumers.
- Firehose: fully managed, near real time delivery to S3, Redshift, OpenSearch Service and similar targets.
- MSK is managed Kafka, for teams that need Kafka compatibility.
- DataSync automates online transfer from on premises file systems to AWS.
- Snow Family devices move data offline when bandwidth is limited, and can compute at the edge.
- Transfer Family provides managed SFTP, FTPS and FTP access to S3 and EFS.
- Storage Gateway bridges on premises applications to AWS storage.
- DMS migrates databases with minimal downtime using continuous replication.
Common mistakes
Using Data Streams when Firehose is enough. Data Streams needs consumers and shard planning. Firehose removes that work for simple delivery.
Expecting Firehose to replay. Firehose delivers to a destination and does not keep a replayable log. Use Data Streams for replay.
Pushing petabytes over a small link. Do the math. A physical device can finish sooner than a long online copy.
Confusing DataSync with Storage Gateway. DataSync moves data on a schedule. Storage Gateway gives ongoing local access to cloud backed storage.
A single hot partition key sends all records to one shard and limits throughput. Choose a key with many distinct values.
Check yourself
Which service loads streaming data into S3 with no consumer code?
Amazon Data Firehose.
When is a Snowball Edge device a better choice than an online transfer?
When the data set is very large, or the network is slow or unreliable, so shipping a device is faster.
Which service gives partners SFTP access to files stored in S3?
AWS Transfer Family.
Topic 6 of 7 in Design High Performing Architectures
Test yourself on Design High Performing Architectures
The full AWS SAA-C03 question bank has 823 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.