Troubleshooting Detection Pipelines

What it is

A detection pipeline has several stages: a source produces data, a log or event service collects it, a filter or rule evaluates it, and a notification reaches a person. The most dangerous failure is silence, because nothing tells you that the alert path is broken.

Troubleshooting means finding which stage stopped, and the exam rewards a method: start at the source and move one stage at a time toward the person.

How it works

Step 1: is the source producing data

Check that the service is on in the right account and Region. GuardDuty, Security Hub, Config and Inspector are all Regional, and member accounts must be enrolled with the delegated administrator.

For CloudTrail, confirm the trail is logging and look at its delivery errors.

# AWS CLI
aws cloudtrail get-trail-status --name org-trail

The output shows IsLogging, the time of the last delivery, and LatestDeliveryError when the bucket or key rejects delivery.

Step 2: is the data being delivered

Flow logs show a log status of NODATA when no traffic occurred and SKIPDATA when some records were skipped. Neither is an error in the log configuration.

Step 3: does the filter or rule match

Metric filters and event patterns fail quietly when the pattern is wrong.

• Test a metric filter pattern before saving it with aws logs test-metric-filter, using sample log lines.
• Remember that a metric filter only sees events that arrive after it exists.
• EventBridge patterns are case sensitive and match whole values. A rule for IAM changes must be in us-east-1.
• Check the rule is enabled, and that it is on the same event bus that receives the event.
• Look at the TriggeredRules and Invocations metrics in the AWS/Events namespace to see whether the rule fires.

Step 4: can the action be called

When a rule matches but nothing happens, look at the target.

The FailedInvocations metric rises when EventBridge cannot deliver. A dead letter queue on the target keeps the failed events so that you can inspect them.

For alarms, read the alarm history.

# AWS CLI
aws cloudwatch describe-alarm-history --alarm-name RootAccountUsage \
  --history-item-type Action

If the history shows a failed action, the alarm worked and the notification path failed. Also check that alarm actions are enabled, because a disabled action never runs.

Step 5: why the finding is missing

• GuardDuty may have created the finding and a suppression rule archived it. Look at archived findings.
• Security Hub control results depend on AWS Config recording. Without it, controls show no data.
• A cross Region aggregator only links Regions you choose, and the origin Region still needs the service enabled.
• Newly created accounts are not covered unless auto enable is set for the organization.

Using the tools

Use CloudTrail to see whether the API call happened and who made it. Use AccessDenied events in CloudTrail to see which permission blocked a service role, such as a Lambda role that cannot write to a log group. Use VPC Reachability Analyzer or flow logs only when the problem is network related.