Monitoring Data Pipelines

What it is

A pipeline that runs without anyone watching will eventually fail. Monitoring tells you when, where and why, ideally before users notice.

AWS monitoring has three pillars: metrics (numbers over time), logs (detailed text records) and events or traces (what happened and in what order).

How it works

Amazon CloudWatch metrics and alarms

Most data services publish metrics to CloudWatch automatically. An alarm watches a metric against a threshold and changes state, for example to ALARM. An alarm can send a notification through Amazon SNS or start an Auto Scaling action, and an SNS topic can in turn invoke a Lambda function.

Useful metrics by service:

Iterator age is the classic stream signal. If it grows, consumers are falling behind the producers, so add consumers, shards or batch capacity.

CloudWatch Logs

Glue, Lambda, MWAA, EMR and others write logs to CloudWatch Logs, organized into log groups and streams. Set a retention period on each log group, because the default keeps logs forever and costs grow. Logs Insights runs queries over log data, and metric filters turn log patterns into metrics you can alarm on.

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 20

Events and notifications

Amazon EventBridge receives state change events, such as a Glue job failing, an EMR step ending or a Step Functions execution failing. A rule matches the event and sends it to SNS, Lambda or another target. This is the clean way to alert on job outcomes without polling.

SNS delivers notifications to email, SMS, HTTP endpoints and Lambda. For team chat or ticketing, subscribe a Lambda function or use a supported chat integration.

Auditing activity with CloudTrail

AWS CloudTrail records API calls such as who started a job, changed a table or altered a bucket policy. CloudTrail answers "who did what", while CloudWatch answers "how is it performing". The audit page covers CloudTrail in more detail.

Tracing and workflow visibility

AWS X-Ray traces requests across services such as Lambda and API calls, which helps find slow steps. Step Functions shows each execution graph with the state input, output and error, which is often the fastest way to find which step failed. MWAA exposes Airflow task logs, and Glue offers the Spark UI for job diagnosis.

Service specific monitoring

Amazon Redshift provides system tables and views that show running queries, queue waits and disk use. Athena lists query history with bytes scanned.

Amazon EMR logs to S3 and has cluster metrics. Amazon OpenSearch Service can store and visualize log data for team dashboards.

Designing alarms

Alarm on symptoms people care about: failed runs, late data and growing lag. Avoid alarms on every metric, which trains people to ignore alerts. Use composite alarms to combine signals, and set the missing data handling so a silent pipeline counts as a problem.

Cost and operational hygiene

Set log retention, keep dashboards for key pipelines and review alarms after each incident. Tag resources by pipeline so you can filter metrics and costs. Document runbooks so alerts lead to action.