File Formats and Compression

What it is

The format and layout of files in Amazon S3 decide how fast and how cheaply you can query them. A good transformation step often does nothing more than convert raw files into a better format.

Data engineers describe a workload by the three Vs. Volume is how much data there is, velocity is how fast it arrives, and variety is how many shapes it has (structured, semi structured and unstructured).

How it works

Row based and columnar formats

A row based format stores each record together. A columnar format stores each column together, so a query that needs three columns out of fifty reads only those three.

Parquet and ORC keep column statistics and apply good compression. Engines use them to skip data that cannot match a filter. These are the default answer when a scenario says "reduce the amount of data scanned" or "analytical queries on S3".

Compression

Compression reduces storage and the bytes read from S3. The trade off is CPU time and whether a file can be split, because a splittable file lets many workers read different parts of one file in parallel.

Columnar formats compress inside the file, so each part can still be read in parallel. A large gzip CSV is a classic bad layout, because one worker has to read the whole file.

File size and partitioning

Too many small files slow every engine, because each file costs a request and some setup time. Too few huge files limit parallelism. Aim for a moderate number of reasonably large files, and compact small ones in a Glue or EMR job.

Partitioning puts files under key prefixes such as year=2026/month=10/. A query that filters on those columns reads only the matching prefixes, which is called partition pruning. Choose partition columns that queries filter on often and that do not produce thousands of tiny partitions.

Converting formats on AWS

Several services can convert formats without custom infrastructure.

• AWS Glue ETL (extract, transform, load) jobs read many formats and write Parquet or ORC.
• Athena CREATE TABLE AS SELECT (CTAS) can write a new table in Parquet with partitions.
• Amazon Data Firehose converts incoming JSON to Parquet or ORC.
• Redshift UNLOAD writes query results to S3 in Parquet.

CREATE TABLE sales_parquet
WITH (format = 'PARQUET', external_location = 's3://my-bucket/curated/sales/',
      partitioned_by = ARRAY['sale_date'])
AS SELECT order_id, amount, sale_date FROM sales_csv;

The partition column must come last in the select list.

Reading the three Vs

High volume points to scalable storage and engines such as S3 with EMR or Glue. High velocity points to Kinesis, MSK or Firehose. High variety points to a schema on read approach, with the Data Catalog describing structure and Glue or EMR handling messy inputs.