Data Operations and Support · Topic 5 of 6

Troubleshooting Data Pipelines

Diagnose common AWS data pipeline failures such as throttling, lag, skew, small files, permissions and network paths using logs, metrics and service tools.

5 min

In a nutshell

Check permissions of the execution role first for AccessDenied errors.
Timeouts and hangs inside a VPC suggest routing, security group or endpoint problems.
Throttling needs backoff, more capacity, a better key or a buffer such as SQS.

What it is

Troubleshooting is the skill of moving from a symptom, such as a late report, to a cause, such as a throttled shard or a missing permission. The exam gives a symptom and asks for the most likely cause or the best fix.

A reliable method is to check recent changes first, then permissions, then network, then capacity, and then data issues.

How it works

A diagnostic order

What changed? A new deployment, schema change, key rotation or role edit often explains a sudden failure.
Is it permissions? Look for AccessDenied in the logs. Check the role used by the job, not your own user.
Is it network? Timeouts and hangs point to subnets, security groups, endpoints and DNS.
Is it capacity? Throttling, lag and memory errors show a limit was reached.
Is it the data? Bad records, schema drift, skew and small files cause failures and slowness.

Common symptoms and causes

Symptom	Likely cause	Typical fix
Producer gets `ProvisionedThroughputExceededException`	Shard limit reached or hot shard	More shards, better partition key, backoff
Kinesis consumer lag grows	Slow consumer	Enhanced fan out, more shards, faster processing
Lambda throttled	Concurrency limit reached	Raise limits, use reserved concurrency, buffer with SQS
Lambda timeout	Work needs more time	More memory, smaller batches, or move to Glue
Glue job out of memory	Skew or large partitions	Larger worker type, repartition, salting
Glue job cannot reach database or S3	Network path or endpoint	Fix subnet route, security group, S3 gateway endpoint
Athena query very slow	Scans too much or many small files	Parquet, partitions, compaction
Athena returns zero rows	Partitions not in the catalog	Run a crawler, `MSCK REPAIR TABLE` or use projection
Redshift `COPY` fails	Bad data, role, or file format	Check `STL_LOAD_ERRORS` or `SYS_LOAD_ERROR_DETAIL`
Redshift queries queue	Too many concurrent queries	WLM, concurrency scaling, or more capacity
DynamoDB throttling	Hot partition or low capacity	Better key, on demand mode, more capacity

Using logs effectively

Start in CloudWatch Logs for the failing run. Use Logs Insights to filter errors across many invocations or jobs.

For Glue, enable continuous logging and the Spark UI, which shows stages, tasks and skew. For EMR, read step logs in S3 and YARN or Spark history. For Step Functions, open the execution to see the failed state and its error message.

For Redshift load errors, query the load error system view.

SELECT query, filename, line_number, colname, err_reason
FROM stl_load_errors
ORDER BY starttime DESC
LIMIT 10;

The err_reason column explains why a row failed, such as a bad type or a value too long.

Data skew and small files

Skew means a few keys hold most of the data, so one worker does far more work than the rest. Signs are one slow task in a Spark stage and memory errors on single workers. Fixes include a better key, salting (adding a random suffix) and broadcasting small tables.

Small files slow Athena, Glue and EMR because each file adds overhead. Compact them by rewriting with a Glue job, and tune producers such as Firehose buffer size to create larger files.

Network and permission checks

For a job in a VPC, verify the route to S3 (gateway endpoint or NAT gateway), the security group rules and DNS settings. For permissions, check the job role policy, the bucket policy, KMS key policy and Lake Formation grants. Remember that all of them must allow the action.

Retry and recovery

After a fix, recover safely. Re run only the failed partition or time window, use idempotent writes, and clear a stuck lock or checkpoint only when you understand it. Dead letter queues hold failed messages, and you can reprocess them after the fix.

Performance tuning habits

Measure first. Compare the run time before and after each change, and change one thing at a time. Prefer fixing data layout (format, partitions, file size) before buying more compute.

What to remember for the exam

Check permissions of the execution role first for AccessDenied errors.
Timeouts and hangs inside a VPC suggest routing, security group or endpoint problems.
Throttling needs backoff, more capacity, a better key or a buffer such as SQS.
Growing iterator age means consumers are behind.
Skew shows as one slow task. The fix is a better key or salting, not just more workers.
Many small files slow queries, so compact them and make producers write larger files.
STL_LOAD_ERRORS explains Redshift COPY failures.
Zero rows in Athena often means missing partitions.

Exam tip

When two answers both add capacity, prefer the one that fixes the cause, such as the partition key or file layout, because the exam favors root cause fixes.

Common mistakes

Adding workers for a skew problem. One hot key still lands on one worker.

Checking your own permissions. The job or function role is what counts.

Ignoring the network path. A correct role cannot help a job that cannot reach S3.

Re running everything. A full re run without idempotent writes can duplicate data.

Watch out

Resetting a checkpoint or bookmark to fix one failure can reprocess old data, so confirm the writes can safely repeat first.

Check yourself

Where do you look first for the reason a Redshift COPY failed?
The load error view, STL_LOAD_ERRORS, which gives the row and reason.
A Glue job in a VPC times out reading S3. What do you check?
The route to S3, such as a gateway endpoint or NAT gateway, and the security group.
Why does adding workers not fix a skewed Spark job?
The hot key still goes to one worker, so the data distribution has to change.

Topic 5 of 6 in Data Operations and Support

Test yourself on Data Operations and Support

The full AWS DEA-C01 question bank has 845 questions, with timed practice exams and an explanation for every answer.

Subscribe Try Sample Exam

This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.