Ingesting Cloud Logging and Monitoring Data into Splunk

What it is

Splunk collects logs from many sources into one searchable place, so analysts can correlate cloud and security events.

Cloud providers and security products each produce their own logs. Splunk brings them together by receiving, parsing and storing them as events. The exam tests how data gets in, and how you check that it arrived correctly.

How it works

First choose the input method for the source, then set the index and sourcetype, and finally verify.

Splunk building blocks

A correct sourcetype matters. It decides field extraction and timestamps, so a wrong one breaks searches and dashboards.

Input methods for cloud data

Different sources suit different methods.

• Cloud provider add-ons: pull logs from the cloud APIs or storage.
• HTTP Event Collector: receives events pushed by an application or a cloud function.
• Syslog: receives messages from firewalls and appliances, through a syslog server or a forwarder.
• Streaming services: cloud queues or event hubs that carry logs to Splunk.
• Forwarders: collect files and logs from servers.

Cloud provider add-ons

Splunk offers add-ons for the large providers. The Splunk Add-on for AWS can collect data such as CloudTrail audit events, VPC flow logs, GuardDuty findings and S3 based logs. The Microsoft Cloud Services add-on collects Azure data using event hubs, and the Google Cloud Platform add-on collects logs through Pub/Sub.

To configure one, you install the add-on and create an account or credentials with read only access to the log source. You then create an input, choose the index and set the interval. Use a dedicated least privilege role, so the connection cannot change anything in the cloud.

Splunk Data Manager and the Cisco Security Cloud app can also help to onboard data from supported sources.

HTTP Event Collector

HEC accepts events over HTTPS. You enable it in Splunk, create a token, and assign an index and a default sourcetype. A sender then posts JSON with the token in an authorization header.

Authorization: Splunk TOKEN_VALUE

{
  "time": 1700000000,
  "host": "fw1",
  "source": "cloud_fw",
  "sourcetype": "my_firewall_logs",
  "index": "security",
  "event": {"action": "block", "src": "10.1.1.5", "dst": "203.0.113.9"}
}

HEC listens on port 8088 by default, and the event endpoint is /services/collector/event. Use HTTPS and one token per source, so you can revoke a token without affecting others.

Syslog from security products

Firewalls and other security tools can send syslog. A common design is a syslog server or Splunk Connect for Syslog that receives the messages and passes them to Splunk. This avoids losing data when Splunk restarts, and makes it easy to tag each source with the right sourcetype.

Verifying ingestion

After you configure an input, check that data is arriving and parsed. A quick search shows events, their source and their time.

index=aws sourcetype=aws:cloudtrail eventName=ConsoleLogin
| stats count by userIdentity.arn, sourceIPAddress

index=security sourcetype=aws:cloudwatchlogs:vpcflow
| stats sum(bytes) by src, dest

Typical problems include a wrong index, which means searches find nothing, a wrong sourcetype, which means fields are missing, and a wrong time zone, which means events appear at the wrong time. A blocked network path, an expired token, or insufficient permissions on the cloud account can also stop the data.

Using the data

Once the data is in, you can build searches, dashboards and alerts. The Common Information Model (CIM) maps different log formats to common field names, so one search can cover many products. Alerts can then trigger notifications or automation.