AWS Glue ETL Jobs

What it is

AWS Glue is a serverless data integration service. Its ETL (extract, transform, load) engine runs your code on managed infrastructure and you pay for the capacity while a job runs.

A Glue job is a script plus settings: the role it runs as, the type and number of workers, the connections it needs and its parameters. You can write the script yourself or build it visually in AWS Glue Studio, which generates PySpark code.

How it works

Job types

Spark jobs run on workers. Capacity is measured in DPUs (data processing units), and the worker type (such as G.1X or G.2X) sets memory and CPU for each worker. Glue can also scale the number of workers automatically.

DynamicFrames and DataFrames

A DynamicFrame is a Glue specific structure that handles records with inconsistent schemas. A field may hold different types in different rows, and a DynamicFrame can represent that as a choice type, which you then resolve with resolveChoice. You can convert to and from a Spark DataFrame when you need standard Spark functions.

datasource = glueContext.create_dynamic_frame.from_catalog(
    database="sales_db", table_name="raw_orders",
    transformation_ctx="datasource")
mapped = ApplyMapping.apply(frame=datasource, mappings=[
    ("order_id", "string", "order_id", "string"),
    ("amount", "string", "amount", "double")])
glueContext.write_dynamic_frame.from_options(
    frame=mapped, connection_type="s3",
    connection_options={"path": "s3://my-bucket/curated/orders/"},
    format="parquet")

The transformation_ctx argument is what job bookmarks use to track state for that step.

Job bookmarks

A job bookmark remembers what the job already processed, so the next run reads only new data. This gives incremental loads without custom code. You can pause or reset a bookmark when you want to reprocess.

Reading and writing data

Glue reads from S3, JDBC databases, DynamoDB, Kinesis, MSK and many other sources. For databases, a Glue connection stores the network and credential details. When the database is in a VPC (virtual private cloud), the job runs with a connection that places it in a subnet with a security group, and that subnet needs a route to S3, often through a gateway endpoint.

Pushdown predicates limit which partitions are read, so a job filters on partition columns before loading data.

glueContext.create_dynamic_frame.from_catalog(
    database="sales_db", table_name="raw_orders",
    push_down_predicate="year == '2026' and month == '10'")

Triggers, parameters and security

Jobs start from the console, a schedule, a Glue trigger, EventBridge or a workflow.

Job parameters let the same script run for different dates or paths. A Glue job needs an IAM (Identity and Access Management) role that can read the sources, write the targets and read the script in S3. Database passwords belong in AWS Secrets Manager, not in the script.

Tuning and common failures

Monitor jobs with CloudWatch metrics and logs, and the Spark UI when enabled. Glue also offers automatic retries on a job.

Glue and other tools

Glue DataBrew is a separate visual data preparation tool with built in transforms. Glue Data Quality adds rules to a job. Both are covered in the data operations chapter.