Logs Explorer, Log Routing and Retention, and Redacting Sensitive Data

What it is

Logs are only useful if you can find the right entries, keep them for as long as rules require, send them to the tools that need them, and make sure they do not leak sensitive data.

Cloud Logging provides the Logs Explorer for searching, sinks for routing, log buckets for retention, and controls for who sees what.

How it works

Logs Explorer and the Logging query language

The Logs Explorer searches log entries with the Logging query language. Lines are joined with AND by default.

resource.type="cloud_run_revision"
resource.labels.service_name="checkout"
severity>=ERROR
jsonPayload.message:"payment"

Use the histogram to spot when errors began, and Log Analytics with SQL when you need grouping and aggregation across large volumes, such as errors per service per hour.

Exporting and retaining logs

Every project has two buckets: _Required (Admin Activity, System Event and some other audit logs, kept 400 days, cannot be changed or deleted) and _Default (everything else, 30 days by default, adjustable). You can create custom log buckets with their own region and retention.

gcloud logging sinks create audit-to-bq \
  bigquery.googleapis.com/projects/sec-logs/datasets/audit_logs \
  --log-filter='logName:"cloudaudit.googleapis.com"' --use-partitioned-tables

Grant the sink's writer identity access on the destination. For compliance, lock a log bucket's retention so it cannot be shortened, or use a Cloud Storage bucket with a locked retention policy.

Handling sensitive data

The best control is never logging personally identifiable information (PII) or protected health information (PHI) at all. When it cannot be avoided, remove it before it is stored.

Data Access audit logs and request logs often contain identifiers. Review them before routing to wide-access tools.

Gemini Cloud Assist for log analysis

In the Logs Explorer, Gemini Cloud Assist can explain a log entry in plain language, such as what an error means and possible next steps. It can also help investigate an issue by pulling together related logs, metrics and recent changes.

Treat its output as a starting point. Confirm with the actual entries and metrics before acting.

A worked example

A health provider must keep audit logs for 7 years, analyze application errors with SQL, and keep patient identifiers out of logs. The engineer creates an aggregated sink of audit logs to a Cloud Storage bucket with a locked 7 year retention policy.

Application logs route to a central log bucket upgraded for Log Analytics, with log views per team. An OpenTelemetry collector processor removes the patient ID field from application logs before export.

How the exam asks about this

• "Find every log entry for one failing request across services." Filter on its trace value in the Logs Explorer.
• "Stream logs in near real time to a third-party SIEM." Create a sink to Pub/Sub and let the SIEM subscribe.
• "Credit card numbers sometimes appear in application logs and must never be stored." Remove them at the source or with a processor in the collection agent before ingestion.