Implementing observability practices and troubleshooting issues · Topic 1 of 6
Collecting Logs with Ops Agent and OpenTelemetry, and Cutting Log Cost
Collect logs with Ops Agent, OpenTelemetry, Cloud Audit Logs, VPC Flow Logs and Cloud Service Mesh, then cut cost with exclusions, sampling and filters.
- 5 min
- Ops Agent for VMs; automatic collection for GKE, Cloud Run and App Engine; OpenTelemetry for anything, anywhere.
- Admin Activity audit logs are always on; Data Access is mostly off by default.
- VPC Flow Logs are per subnet with sampling and aggregation settings.
What it is
Cloud Logging stores logs from Google Cloud services, from your applications and from systems outside Google Cloud. Some logs arrive automatically, some need an agent or instrumentation, and some must be turned on.
Collecting everything at full detail is expensive and noisy. The second half of this topic is reducing volume at the source and at the Log Router without losing what you need.
How it works
Where logs come from
| Source | How logs are collected |
|---|---|
| Compute Engine VMs | Install the Ops Agent; it collects system and application log files and sends them to Cloud Logging |
| GKE | Container stdout and stderr and system logs are collected by GKE's logging; choose system only or system and workloads |
| Cloud Run, Cloud Run functions, App Engine | stdout, stderr and request logs collected automatically |
| Apps anywhere | OpenTelemetry SDKs and collectors, or the Cloud Logging client libraries |
| Google Cloud APIs | Cloud Audit Logs |
| Networks | VPC Flow Logs, firewall rules logging, load balancer logs |
| Service mesh | Cloud Service Mesh proxies produce access logs and telemetry for traffic between services |
Write logs as structured JSON to stdout on GKE and Cloud Run. Fields such as severity and message are recognized, and the rest land in jsonPayload, ready to filter.
Ops Agent
The Ops Agent collects both logs and metrics from Linux and Windows VMs. It is configured in /etc/google-cloud-ops-agent/config.yaml with receivers and pipelines:
logging:
receivers:
app_log:
type: files
include_paths: [/var/log/myapp/*.log]
service:
pipelines:
app_pipeline:
receivers: [app_log]Install it on many VMs with an agent policy or in the VM image. The VM's service account needs the Logs Writer and Monitoring Metric Writer roles.
OpenTelemetry
OpenTelemetry (OTel) is the vendor-neutral standard for logs, metrics and traces. Instrument the app with an OTel SDK and send data to an OTel collector, which exports to Google Cloud. It suits hybrid and multi-cloud estates because the same instrumentation works everywhere.
Cloud Audit Logs
| Type | Enabled | Records |
|---|---|---|
| Admin Activity | Always, cannot be disabled | Configuration changes and metadata writes |
| Data Access | Off by default except BigQuery; turn on per service | Reads of configuration or data, and user data writes |
| System Event | Always | Actions taken by Google systems, such as live migration |
| Policy Denied | Always generated | Access denied by a security policy, such as VPC Service Controls |
Data Access logs can be very large. Enable them only for the services and projects where you need them.
VPC Flow Logs
VPC Flow Logs record samples of network flows to and from VM interfaces, usually configured per subnet (newer configurations can also cover a whole network, VPN tunnels or Interconnect attachments). You choose the aggregation interval, the sample rate and whether to include metadata, and can add a filter. Higher sampling and shorter intervals mean more detail and more cost.
Optimizing logs
| Technique | Where | Effect |
|---|---|---|
| Lower log level in production | Application | Fewer DEBUG lines at the source |
| Sampling | Source settings, such as load balancer logging sample rate or flow log sample rate | Keep a share of high-volume entries |
| Exclusion filters | Sinks, including _Default | Excluded entries are not stored in that bucket and are not charged for storage there |
| Sampled exclusion | sample() in an exclusion filter | Keep a percentage of noisy logs |
| Retention | Per log bucket | Shorter retention for verbose logs |
| Route cold logs | Sink to Cloud Storage | Cheaper long-term retention |
resource.type="http_load_balancer"
AND httpRequest.status<400
AND sample(insertId, 0.9)Used as an exclusion filter, this drops about 90% of successful load balancer requests while keeping every error.
A worked example
A team's Cloud Logging bill doubled. Log Analytics shows most volume is successful load balancer requests and DEBUG logs from one GKE namespace.
The engineer sets the application log level to INFO, adds the sampled exclusion above to the _Default sink, and lowers the VPC Flow Logs sample rate on busy subnets. Admin Activity audit logs are untouched because they are always kept in _Required.
How the exam asks about this
- "Application log files on Compute Engine VMs do not appear in Cloud Logging." Install and configure the Ops Agent, and check the service account roles.
- "Record who read objects in a sensitive bucket." Enable Data Access audit logs for Cloud Storage in that project.
- "Reduce logging cost while keeping all error entries." Add an exclusion filter, sampled if needed, that matches only non-error entries.
What to remember for the exam
- Ops Agent for VMs; automatic collection for GKE, Cloud Run and App Engine; OpenTelemetry for anything, anywhere.
- Admin Activity audit logs are always on; Data Access is mostly off by default.
- VPC Flow Logs are per subnet with sampling and aggregation settings.
- Reduce at the source first, then use exclusions and
sample()at the Log Router. - Structured JSON logs make filtering, metrics and trace correlation possible.
An exclusion filter on the _Default sink only stops storage in _Default. Logs still pass through the Log Router, so other sinks that match them still receive them.
Common mistakes
Turning on Data Access logs everywhere. Volume and cost explode. Enable them for the services that need an access record.
Writing plain text logs. Fields cannot be filtered or turned into metrics easily. Log JSON with severity and context fields.
Excluding logs needed for audits. Check retention requirements before excluding. Route them to cheap storage instead.
Check yourself
Which audit log type cannot be disabled?
Admin Activity audit logs, which record configuration changes and are always written.
What does the Ops Agent collect?
Logs and metrics from Compute Engine VMs, configured with receivers, processors and pipelines in its config file.
How can you keep 10% of noisy successful requests in logs?
Add an exclusion filter that matches those entries with sample(insertId, 0.9), so about 90% are excluded and 10% kept.
Topic 1 of 6 in Implementing observability practices and troubleshooting issues
Test yourself on Implementing observability practices and troubleshooting issues
The full Google Cloud Professional Cloud DevOps Engineer question bank has 751 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.