Network Logs, Cloud IDS and a Secure Logging Strategy

What it is

Network logs show who talked to whom, which firewall rule decided it, and whether the traffic looked malicious. A logging strategy decides which logs to collect, where to keep them, for how long, and who may read them.

How it works

Network log sources

gcloud compute networks subnets update app-subnet --region=europe-west1 \
  --enable-flow-logs --logging-aggregation-interval=interval-5-sec \
  --logging-flow-sampling=0.5

gcloud compute firewall-rules update allow-ssh-from-iap --enable-logging

Packet Mirroring and Cloud IDS

Packet Mirroring clones traffic of chosen VMs to a collector group behind an internal passthrough Network Load Balancer, for deep packet inspection by your own tools. Cloud IDS is a managed intrusion detection service built on Palo Alto Networks threat technology. It receives mirrored traffic at an IDS endpoint and writes threat logs to Cloud Logging.

Analyzing logs with Log Analytics

Upgrade a log bucket to use Log Analytics and query logs with SQL. A linked BigQuery dataset lets you join logs with other data.

SELECT
  JSON_VALUE(json_payload.connection.src_ip) AS src_ip,
  JSON_VALUE(json_payload.connection.dest_ip) AS dest_ip,
  COUNT(*) AS flows
FROM `net-logs.global._Default._AllLogs`
WHERE log_id = 'compute.googleapis.com/vpc_flows'
  AND timestamp > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 HOUR)
GROUP BY src_ip, dest_ip
ORDER BY flows DESC
LIMIT 20;

Designing an effective logging strategy

Turning on everything is expensive and noisy. Turning on too little leaves you blind during an incident.

Designing secure access to logs

Grant log access to groups, and keep the people who administer workloads separate from those who can change or delete the security logs about them.

A worked example

After a suspected breach, the security team needs to know which internal VMs talked to an unknown external address. Flow logs were off in most subnets, so there is no record.

The engineer enables flow logs with 50 percent sampling on all production subnets, firewall logging on deny rules, and Cloud IDS for the database subnet. An aggregated sink sends network and audit logs to a locked bucket in the logging project, upgraded for Log Analytics, and app teams get log views for their own projects only.

How the exam asks about this

• "Identify which firewall rule blocked a connection between two VMs." Turn on Firewall Rules Logging for the relevant rules and check the logs.
• "Detect malware and exploit traffic to sensitive VMs without deploying or managing appliances, alert only." Cloud IDS with Packet Mirroring.
• "Let an application team read logs for its own project from the central log bucket, but not other teams' logs." A log view on the bucket, granted with View Accessor.