Performance Optimization · Topic 6 of 7

Monitoring Streams, Tasks, Pipes and Snowpipe Streaming

Know where to look when a Snowflake pipeline stalls: Snowsight task history, TASK_HISTORY, stale streams, pipe status and Snowpipe Streaming telemetry.

  • 5 min
In a nutshell
  • Snowsight per task Run History covers 7 days; TASK_HISTORY function for recent, ACCOUNT_USAGE for long history.
  • ERROR_INTEGRATION on tasks supports SNS, Event Grid and Pub/Sub; email and webhook come through task alerts.
  • Stale streams must be recreated; SYSTEM$STREAM_HAS_DATA checks for changes.

What it is

A continuous pipeline is a chain of Snowflake objects: ingestion (Snowpipe or Snowpipe Streaming), change tracking (streams), orchestration (tasks), and often dynamic tables. Monitoring means knowing which view or screen belongs to which object, so you can tell whether data is late, stuck or failing.

Snowpipeor Snowpipe Streamingrows landTablechangesStreamconsumed byTask graphwritesTarget table

How it works

Which tool for which object

ObjectWhere to lookTypical question
Pipe (files)SYSTEM$PIPE_STATUS, COPY_HISTORY, PIPE_USAGE_HISTORYIs it running, what is pending, what failed, what did it cost
Snowpipe StreamingEvent table telemetry, offset tokens, serverless usage in metering viewsAre rows arriving, any row or channel errors
StreamSHOW STREAMS, SYSTEM$STREAM_HAS_DATAIs there data, is it stale
TaskSnowsight Task History, TASK_HISTORY, task graph functionsDid it run, fail or skip
Dynamic tableSnowsight, refresh history functions (next page)Is it within target lag

Tasks

In Snowsight, go to Transformation, then Tasks. The account level view lists task runs and can be filtered by status (Succeeded, Failed, Cancelled, Skipped), database, schema and name. The Task Graphs tab shows each graph with its recent run statuses and a duration trend, and a graph page shows each task's predecessors, warehouse and owner.

  • A single task's Run History in Snowsight covers runs from the last 7 days; the account level Task Runs view has a date range filter reaching back up to 12 months.
  • Account level views can lag by about 45 minutes.
  • Viewing history needs OWNERSHIP, MONITOR or OPERATE on the task, the global MONITOR EXECUTION privilege, or ACCOUNTADMIN.

In SQL:

SELECT name, state, error_message, scheduled_time, completed_time
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
  SCHEDULED_TIME_RANGE_START => DATEADD('hour', -24, CURRENT_TIMESTAMP()),
  RESULT_LIMIT => 100))
WHERE state = 'FAILED';
  • INFORMATION_SCHEMA.TASK_HISTORY is the table function for recent runs; the ACCOUNT_USAGE.TASK_HISTORY view keeps longer history with latency.
  • COMPLETE_TASK_GRAPHS and CURRENT_TASK_GRAPHS show finished and running task graph runs.
  • SHOW TASKS shows state (started or suspended), schedule, owner and predecessors.
  • Serverless task cost is tracked in the serverless task history view, and in METERING_HISTORY.

Failure handling:

  • Set ERROR_INTEGRATION on a task to send a message to Amazon SNS, Azure Event Grid or Google Pub/Sub when a run fails. It does not accept email or webhook integrations.
  • Task alerts send an email or webhook when the error rate passes a threshold. They rely on task events in the event table, so set LOG_EVENT_LEVEL on the monitored tasks; at the default of OFF no events are recorded and the alert has nothing to act on.
  • SUSPEND_TASK_AFTER_NUM_FAILURES suspends a task after that many consecutive failures, so a broken task stops burning credits.
  • A task skipped because the previous run was still going shows as Skipped, not Failed.
Exam tip

If a new task never ran, check state first. Tasks are created suspended and need ALTER TASK ... RESUME.

Streams

A stream is healthy when it is consumed regularly.

  • SHOW STREAMS and DESCRIBE STREAM show the STALE flag and STALE_AFTER time.
  • A stream goes stale when its offset falls outside the source table's retention. Snowflake extends retention for an unconsumed stream up to MAX_DATA_EXTENSION_TIME_IN_DAYS (default 14), and a stale stream must be recreated.
  • SYSTEM$STREAM_HAS_DATA('stream_name') returns TRUE when the stream holds changes; tasks use it in a WHEN clause so they skip empty runs.
  • A plain SELECT does not advance the offset, so monitoring queries are safe.
Task runs MERGEconsumesStreamoffset advancesif skipped for longStaleretention passedfixRecreate stream and reload

Pipes

For file based Snowpipe, SYSTEM$PIPE_STATUS('pipe') returns JSON with the execution state and the count of pending files. COPY_HISTORY shows per file load results and errors, and PIPE_USAGE_HISTORY shows credits and files loaded. Pipes can also use an error notification integration to push failures to a queue.

Snowpipe Streaming

There are no files or stages to inspect, so monitoring relies on other signals.

  • Event table telemetry records ingestion progress, processing time, and row and channel errors.
  • Offset tokens are stored by the client with each batch. After a restart or error, the last committed offset token tells the client where to resume, with no gaps or duplicates.
  • Cost appears in the metering views, such as METERING_HISTORY.
Watch out

If the client does not track committed offset tokens, you cannot prove nothing was lost or duplicated after a failure. Treat the offset token as the source of truth for resuming.

Common mistakes

Assuming ACCOUNT_USAGE.TASK_HISTORY shows the last minute. It lags. Use the Information Schema table function for recent runs.

Alerting on tasks without event logging. With LOG_EVENT_LEVEL at its default, task alerts see no data.

Selecting from a stream and expecting it to clear. Only committed DML that consumes the stream advances the offset.

Looking for Snowpipe Streaming in COPY_HISTORY. It uses channels, not files; check event table telemetry and offset tokens.

What to remember for the exam

  • Snowsight per task Run History covers 7 days; TASK_HISTORY function for recent, ACCOUNT_USAGE for long history.
  • ERROR_INTEGRATION on tasks supports SNS, Event Grid and Pub/Sub; email and webhook come through task alerts.
  • Stale streams must be recreated; SYSTEM$STREAM_HAS_DATA checks for changes.
  • Pipes: SYSTEM$PIPE_STATUS, COPY_HISTORY, PIPE_USAGE_HISTORY. Streaming: event tables and offset tokens.
  • New tasks are suspended; SUSPEND_TASK_AFTER_NUM_FAILURES stops repeated failures.

Check yourself

  • Which SQL function returns TRUE when a stream has pending changes?

    SYSTEM$STREAM_HAS_DATA('stream_name').

  • What does a Snowpipe Streaming client use to resume safely after a failure?

    The last committed offset token it stored for the channel.

  • Which integration types can a task ERROR_INTEGRATION use?

    Cloud queue and topic services: Amazon SNS, Azure Event Grid and Google Pub/Sub, not email or webhook.

Topic 6 of 7 in Performance Optimization

Test yourself on Performance Optimization

The full Snowflake DEA-C02 question bank has 709 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Snowflake. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.