Step Functions and Glue Workflows

What it is

Orchestration means coordinating the steps of a pipeline: what runs first, what runs in parallel, what happens on failure and when the pipeline is done. A pipeline without orchestration is a set of jobs that someone starts by hand.

AWS offers several orchestrators. This page covers the two most tied to AWS services: AWS Step Functions and AWS Glue workflows.

How it works

AWS Step Functions

Step Functions runs a state machine, which is a workflow written in Amazon States Language (ASL), a JSON based format. You design it visually in Workflow Studio or write the JSON yourself.

Step Functions has direct service integrations with Lambda, Glue, Athena, EMR, ECS (Elastic Container Service), SQS, SNS, DynamoDB and many others. Adding .sync to a Glue or EMR task makes the state machine wait until the job finishes, instead of moving on once the job starts.

{
  "StartAt": "RunGlueJob",
  "States": {
    "RunGlueJob": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": { "JobName": "curate-orders" },
      "Retry": [{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 60,
                  "MaxAttempts": 2, "BackoffRate": 2.0 }],
      "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "NotifyFailure" }],
      "Next": "Done"
    },
    "NotifyFailure": { "Type": "Task",
      "Resource": "arn:aws:states:::sns:publish",
      "Parameters": { "TopicArn": "arn:aws:sns:us-east-1:123456789012:etl-alerts",
                      "Message": "Glue job failed" },
      "End": true },
    "Done": { "Type": "Succeed" }
  }
}

Retry and Catch

Retry repeats a failed state with an interval and a backoff rate. Catch sends the run to a fallback state once retries are used up. Together they give resilient pipelines without custom code.

Standard and Express workflows

For long ETL pipelines, choose Standard. The Map state has a distributed mode that can process large numbers of S3 objects in parallel child workflows.

Waiting for outside work

A callback pattern uses a task token. The state machine pauses at a task, passes the token to another system, and resumes when that system returns the token with a success or failure call. This suits human approval or a job that runs outside AWS.

AWS Glue workflows

A Glue workflow chains Glue triggers, crawlers and jobs in one graph that you can view and monitor. A trigger can be on demand, on a schedule or conditional on the success or failure of earlier jobs and crawlers. Workflows are the simplest option when the whole pipeline is made of Glue parts.

Choosing between them