Investigating Security Events

What it is

Investigation answers four questions about an event: what happened, who or what did it, which resources and data were affected, and whether it is over. The answers come from correlating logs and findings, not from any one tool.

Good investigations start narrow, from the indicator you have, and widen: from one key to all its activity, from one instance to the network it talked to.

How it works

Choose the right data source

Querying CloudTrail with Athena

For history beyond 90 days or for data events, query the trail files in S3. An Athena table over the trail location lets you use SQL.

-- Athena: everything one access key did, oldest first
SELECT eventtime, eventsource, eventname, awsregion,
       sourceipaddress, useragent, errorcode
FROM cloudtrail_logs
WHERE useridentity.accesskeyid = 'AKIAIOSFODNN7EXAMPLE'
  AND eventtime > '2026-09-25T00:00:00Z'
ORDER BY eventtime;

Filter on partitions such as the date and Region where your table allows it, because Athena charges by data scanned. Useful pivots are sourceipaddress, useragent (a script user agent from a person who normally uses the console is a clue), errorcode and eventname.

CloudTrail Lake offers the same style of SQL on an event data store without building tables, and it can query events from several accounts when the store is organization wide. CloudWatch Logs Insights is the quick option when the trail also feeds a log group.

Reading the clues

• A burst of AccessDenied errors from a new IP suggests an attacker probing the permissions of a stolen key.
• Calls such as GetCallerIdentity, ListBuckets and DescribeInstances early in a session suggest reconnaissance.
• StopLogging, DeleteTrail or DeleteDetector (GuardDuty) suggest defence evasion.
• RunInstances in unusual Regions suggests resource abuse such as mining.
• Console sign in without MFA, or from a new country, suggests a stolen password.

The userIdentity block tells you the principal type, the role session name, and, for assumed roles, the original identity through the session context. Follow an assumed role session back to the person or system that assumed it.

Using Detective

Detective builds a behaviour graph from CloudTrail, flow logs, GuardDuty and other sources. From a finding, you can open the principal and see its API call volume over time, new geographies and the resources it touched compared with its baseline. This is faster than writing queries when you need a timeline across accounts.

Network investigation

Flow logs give source, destination, port, bytes and the accept or reject action. Find the compromised host, list its outbound destinations, and look for unusual ports or large transfers to a single address. Match any suspect domain against Resolver query logs, since DNS often reveals command and control traffic that has no distinct port.

For packet level proof, Traffic Mirroring can copy traffic from a network interface to an analysis tool, but it must be enabled before the traffic occurs.

Post incident work

Close the loop. Record the timeline, root cause, impact and time to detect, contain and recover. Then change something durable.

• Add a detection for the gap, such as an alarm or a GuardDuty response rule.
• Remove the permission or exposure that allowed the incident, perhaps with a permission boundary or an SCP.
• Update the runbook and run the scenario again as a test.
• If regulations apply, notify the right parties within the required period.