Architecture Design and Implementation Built for the Cloud · Topic 10 of 11
Data Lakes, Analytics Pipelines and Data Sharing
Design S3 data lakes and Iceberg lakehouses with Lake Formation, pick EMR, Redshift, MSK or AppFlow pipelines, and share data with Glue and Clean Rooms.
- 5 min
- S3 is the lake storage; Parquet plus partitions keeps query cost down.
- Iceberg brings transactions, updates, deletes and time travel to the lake.
- Lake Formation grants table, column, row and cell permissions, and LF-Tags scale those grants.
What it is
A data lake stores raw and processed data in Amazon S3 in open formats, so many engines can query the same copy. A lakehouse adds table features such as transactions and schema changes on top of the lake, usually with Apache Iceberg.
Pipelines feed the lake in real time or in batches. Catalogs and permissions then decide who in which account can see which tables, columns and rows.
How it works
Lake and lakehouse building blocks
- Zones in S3: raw (as received), cleaned, and curated for consumers. Keep raw data so you can reprocess it.
- Columnar formats: Parquet or ORC with partitioning cut scan cost for Athena and Redshift Spectrum.
- Apache Iceberg tables add ACID transactions, schema evolution, hidden partitioning and time travel to S3 data. Athena, EMR, Glue and Redshift can read and write them.
- Amazon S3 Tables store Iceberg tables in table buckets with automatic maintenance such as compaction.
- Redshift Spectrum lets Redshift query data in S3 through external tables, so the warehouse joins hot local data with cold lake data without loading it.
Choose Iceberg when the lake needs row-level updates and deletes, for example to apply change data capture or to erase a customer's records.
Governing the lake with Lake Formation
AWS Lake Formation sits on the Glue Data Catalog and grants permissions on databases, tables, columns, rows and cells through data filters. LF-Tags let you grant access by tag, such as classification=confidential, instead of table by table.
Integrated engines such as Athena, Redshift Spectrum, EMR and Glue enforce these permissions, which replaces complex S3 bucket policies per dataset.
Real-time and batch analytics
| Need | Choose |
|---|---|
| Kafka compatibility, existing Kafka clients and connectors | Amazon MSK (provisioned or serverless), with MSK Connect |
| AWS-native streaming with simple scaling | Kinesis Data Streams |
| Stateful stream processing, windows and joins in real time | Amazon Managed Service for Apache Flink |
| Large batch processing with Spark, Hive or Presto | Amazon EMR on EC2, on EKS or EMR Serverless |
| Data warehouse for BI and complex SQL | Amazon Redshift, provisioned or serverless |
| SaaS data (such as Salesforce or ServiceNow) into S3 or Redshift without code | Amazon AppFlow |
| Ad hoc SQL on the lake with nothing to manage | Amazon Athena |
Redshift can also ingest from Kinesis and MSK directly with streaming ingestion. Zero-ETL integrations, such as Aurora to Redshift, replicate operational data into the warehouse without building pipelines.
Cataloging and cross-account sharing
| Pattern | How it works | Use it when |
|---|---|---|
| Central Glue Data Catalog with Lake Formation | Producer accounts register data; a central governance account grants consumer accounts access | Data mesh or central lake across many accounts |
| Lake Formation cross-account grants | Share tables with accounts or the organization; consumers query through resource links | Consumers need governed, column or row-level access |
| Redshift data sharing | Share live warehouse data with other clusters or accounts without copying | Several teams need the same warehouse data |
| AWS Clean Rooms | Partners analyze combined datasets under analysis rules, without exposing each other's raw data | Joint analysis with an external party, such as advertising measurement |
Glue crawlers discover schemas and keep the catalog current. A resource link in the consumer account points to the shared database or table, so local engines can query it.
How the exam asks about this
- "Analysts in ten accounts need column-level access to central lake tables with the least policy management" -> Lake Formation with LF-Tags and cross-account grants on a central Glue Data Catalog.
- "Two companies must measure campaign overlap without sharing customer records" -> AWS Clean Rooms with analysis rules.
- "Existing Kafka producers must move to AWS with no client changes" -> Amazon MSK.
What to remember for the exam
- S3 is the lake storage; Parquet plus partitions keeps query cost down.
- Iceberg brings transactions, updates, deletes and time travel to the lake.
- Lake Formation grants table, column, row and cell permissions, and LF-Tags scale those grants.
- MSK for Kafka compatibility, Managed Flink for stateful stream processing, EMR for big batch jobs, AppFlow for SaaS sources.
- Redshift Spectrum queries S3 from Redshift; Redshift data sharing shares live data without copies.
- Clean Rooms enables joint analysis without exposing raw data between parties.
Granting S3 bucket access directly to analysts bypasses Lake Formation's column and row controls. When the scenario asks for fine-grained governed access, the answer goes through Lake Formation permissions, not bucket policies.
Common mistakes
Copying data into every consumer account. Copies drift, cost storage and multiply the governance effort. Share through Lake Formation or Redshift data sharing instead.
Using plain Parquet files when records must be updated. Rewriting partitions by hand is slow and error prone. Iceberg tables support row-level updates and deletes.
Building custom SaaS connectors. Scripts that call SaaS APIs on a schedule add code to maintain. AppFlow provides managed flows for supported SaaS applications.
Check yourself
What lets Redshift join warehouse tables with data that stays in S3?
Redshift Spectrum, which queries S3 data through external tables in the data catalog.
How do LF-Tags simplify data lake permissions?
You grant access to tag values, and every table or column with that tag inherits the grant, instead of granting each resource one by one.
Which service lets two organizations analyze combined data without sharing raw records?
AWS Clean Rooms, which runs queries under analysis rules agreed by the collaborators.
Topic 10 of 11 in Architecture Design and Implementation Built for the Cloud
Test yourself on Architecture Design and Implementation Built for the Cloud
The full AWS SAP-C03 question bank has 850 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.