Choosing a Processing Service

What it is

Many AWS services can transform data. The exam gives you a scenario with data size, skills, cost goals and operational effort, and asks which one fits best.

The core question is how much control you need compared with how much infrastructure you want to manage. Serverless services reduce operations, while cluster based services give more tuning and framework choice.

How it works

The main candidates

Glue is the default answer for "serverless ETL". EMR is the answer when the scenario needs a specific framework or version, custom libraries at the cluster level, very large long running jobs, or fine control over instances.

Lambda in transformation

Lambda suits small, bursty work, such as converting a file that has just landed in S3. A function can run for at most 15 minutes, so large jobs do not fit. Memory and the temporary /tmp space are configurable up to 10 GB, which helps with moderate files.

Amazon EMR deployment options

EMR comes in three forms.

A classic EMR cluster has a primary node that coordinates, core nodes that run tasks and store HDFS (Hadoop Distributed File System) data, and optional task nodes that only compute. Task nodes are a good place for Spot Instances, because losing one does not lose stored data.

For cost, use transient clusters that start, run a job and terminate, with data kept in S3 through EMRFS (the EMR file system) instead of HDFS. A long running cluster fits interactive work or constant load.

Spark and SQL transforms

Glue and EMR both run Apache Spark. A job reads data into a distributed dataset, applies transformations such as filters, joins and aggregations in parallel, and writes the result. Redshift handles the same shapes in SQL, and Athena does it on data in place.

Redshift stored procedures and INSERT INTO ... SELECT statements suit an ELT pattern, where raw data is loaded first and transformed inside the warehouse. Spectrum lets Redshift read S3 data in the same query.

Joining and enriching data

Transformations often join streams with reference data or combine files from several sources. In Glue this is a join between two datasets, in Athena it is a SQL join, and in Flink it is a streaming join against a lookup. Small reference tables can be broadcast to workers, which avoids a costly shuffle of the large side.

Cost and scaling levers

Consider the pricing model as well as the feature fit.

• Glue bills for the capacity used while a job runs, and the Flex execution class lowers cost for work that is not urgent.
• EMR can use Spot Instances and automatic scaling.
• Athena costs follow bytes scanned, so good formats and partitions save money.
• Lambda costs follow invocations and duration.