S3 Data Lake Design

What it is

An Amazon S3 data lake keeps data in its original and processed forms in buckets, so many engines can read the same files. Storage is cheap and separate from compute, which means Athena, Glue, EMR and Redshift Spectrum can all work on one copy.

The design question is how to lay the data out so it is easy to find, cheap to query and safe to change.

How it works

Zones

Most lakes use layers, sometimes called bronze, silver and gold, or raw, curated and consumption.

Keeping raw data unchanged lets you rebuild later layers if business rules change.

Buckets, prefixes and partitions

S3 has no real folders. A key such as curated/orders/year=2026/month=10/part-0001.parquet is one name, and the part before the last slash acts like a folder. Hive style key=value prefixes let Athena and Glue treat them as partitions.

Choose partition columns that queries filter on, usually a date. Avoid very fine partitions, which create many tiny files. S3 scales request rates per prefix, so spreading data over several prefixes supports parallel access.

Open table formats

Plain files in S3 cannot be updated safely in place. Open table formats such as Apache Iceberg, Apache Hudi and Delta Lake add a metadata layer on top of Parquet files.

They provide ACID transactions, row level inserts, updates and deletes, snapshots for time travel and schema evolution. Athena, Glue, EMR and Redshift support Apache Iceberg tables, and Iceberg is the format the exam mentions most often.

CREATE TABLE orders_iceberg (
  order_id bigint, amount double, order_date date)
PARTITIONED BY (order_date)
LOCATION 's3://my-bucket/curated/orders_iceberg/'
TBLPROPERTIES ('table_type'='ICEBERG');

Use such a table when the lake must support updates, deletes (for example privacy erasure requests) or change data capture merges.

Querying the lake

All of them can share one Glue Data Catalog, so the table definitions are written once.

Controlling access to the data

Bucket policies and IAM policies decide who can read objects. S3 Access Points give each application its own named endpoint and policy, which simplifies a bucket shared by many teams. S3 Block Public Access prevents public exposure at the bucket or account level.

For fine grained, column and row level control across Athena, Redshift and EMR, use AWS Lake Formation instead of managing object permissions by hand.

Performance and cost levers

Convert to columnar formats, compress, compact small files and partition on filter columns. Use S3 Intelligent-Tiering or lifecycle rules for data with changing access, which the lifecycle page covers.

S3 delivers strong read after write consistency for all operations, so a listing or read right after a write sees the new object. This removes the need for older workarounds that some tutorials still describe.

Choosing lake or warehouse

Keep cold, large and varied data in the lake. Load hot, frequently queried, modelled data into Redshift for speed and concurrency. Many designs do both and use Spectrum to join them.