Performance Optimization · Topic 6 of 7
Monitoring Streams, Tasks, Pipes and Snowpipe Streaming
Know where to look when a Snowflake pipeline stalls: Snowsight task history, TASK_HISTORY, stale streams, pipe status and Snowpipe Streaming telemetry.
- 5 min
- Snowsight per task Run History covers 7 days;
TASK_HISTORYfunction for recent,ACCOUNT_USAGEfor long history. ERROR_INTEGRATIONon tasks supports SNS, Event Grid and Pub/Sub; email and webhook come through task alerts.- Stale streams must be recreated;
SYSTEM$STREAM_HAS_DATAchecks for changes.
What it is
A continuous pipeline is a chain of Snowflake objects: ingestion (Snowpipe or Snowpipe Streaming), change tracking (streams), orchestration (tasks), and often dynamic tables. Monitoring means knowing which view or screen belongs to which object, so you can tell whether data is late, stuck or failing.
How it works
Which tool for which object
| Object | Where to look | Typical question |
|---|---|---|
| Pipe (files) | SYSTEM$PIPE_STATUS, COPY_HISTORY, PIPE_USAGE_HISTORY | Is it running, what is pending, what failed, what did it cost |
| Snowpipe Streaming | Event table telemetry, offset tokens, serverless usage in metering views | Are rows arriving, any row or channel errors |
| Stream | SHOW STREAMS, SYSTEM$STREAM_HAS_DATA | Is there data, is it stale |
| Task | Snowsight Task History, TASK_HISTORY, task graph functions | Did it run, fail or skip |
| Dynamic table | Snowsight, refresh history functions (next page) | Is it within target lag |
Tasks
In Snowsight, go to Transformation, then Tasks. The account level view lists task runs and can be filtered by status (Succeeded, Failed, Cancelled, Skipped), database, schema and name. The Task Graphs tab shows each graph with its recent run statuses and a duration trend, and a graph page shows each task's predecessors, warehouse and owner.
- A single task's Run History in Snowsight covers runs from the last 7 days; the account level Task Runs view has a date range filter reaching back up to 12 months.
- Account level views can lag by about 45 minutes.
- Viewing history needs
OWNERSHIP,MONITORorOPERATEon the task, the globalMONITOR EXECUTIONprivilege, orACCOUNTADMIN.
In SQL:
SELECT name, state, error_message, scheduled_time, completed_time
FROM TABLE(INFORMATION_SCHEMA.TASK_HISTORY(
SCHEDULED_TIME_RANGE_START => DATEADD('hour', -24, CURRENT_TIMESTAMP()),
RESULT_LIMIT => 100))
WHERE state = 'FAILED';
INFORMATION_SCHEMA.TASK_HISTORYis the table function for recent runs; theACCOUNT_USAGE.TASK_HISTORYview keeps longer history with latency.COMPLETE_TASK_GRAPHSandCURRENT_TASK_GRAPHSshow finished and running task graph runs.SHOW TASKSshows state (started or suspended), schedule, owner and predecessors.- Serverless task cost is tracked in the serverless task history view, and in
METERING_HISTORY.
Failure handling:
- Set
ERROR_INTEGRATIONon a task to send a message to Amazon SNS, Azure Event Grid or Google Pub/Sub when a run fails. It does not accept email or webhook integrations. - Task alerts send an email or webhook when the error rate passes a threshold. They rely on task events in the event table, so set
LOG_EVENT_LEVELon the monitored tasks; at the default ofOFFno events are recorded and the alert has nothing to act on. SUSPEND_TASK_AFTER_NUM_FAILURESsuspends a task after that many consecutive failures, so a broken task stops burning credits.- A task skipped because the previous run was still going shows as Skipped, not Failed.
If a new task never ran, check state first. Tasks are created suspended and need ALTER TASK ... RESUME.
Streams
A stream is healthy when it is consumed regularly.
SHOW STREAMSandDESCRIBE STREAMshow theSTALEflag andSTALE_AFTERtime.- A stream goes stale when its offset falls outside the source table's retention. Snowflake extends retention for an unconsumed stream up to
MAX_DATA_EXTENSION_TIME_IN_DAYS(default 14), and a stale stream must be recreated. SYSTEM$STREAM_HAS_DATA('stream_name')returns TRUE when the stream holds changes; tasks use it in aWHENclause so they skip empty runs.- A plain
SELECTdoes not advance the offset, so monitoring queries are safe.
Pipes
For file based Snowpipe, SYSTEM$PIPE_STATUS('pipe') returns JSON with the execution state and the count of pending files. COPY_HISTORY shows per file load results and errors, and PIPE_USAGE_HISTORY shows credits and files loaded. Pipes can also use an error notification integration to push failures to a queue.
Snowpipe Streaming
There are no files or stages to inspect, so monitoring relies on other signals.
- Event table telemetry records ingestion progress, processing time, and row and channel errors.
- Offset tokens are stored by the client with each batch. After a restart or error, the last committed offset token tells the client where to resume, with no gaps or duplicates.
- Cost appears in the metering views, such as
METERING_HISTORY.
If the client does not track committed offset tokens, you cannot prove nothing was lost or duplicated after a failure. Treat the offset token as the source of truth for resuming.
Common mistakes
Assuming ACCOUNT_USAGE.TASK_HISTORY shows the last minute. It lags. Use the Information Schema table function for recent runs.
Alerting on tasks without event logging. With LOG_EVENT_LEVEL at its default, task alerts see no data.
Selecting from a stream and expecting it to clear. Only committed DML that consumes the stream advances the offset.
Looking for Snowpipe Streaming in COPY_HISTORY. It uses channels, not files; check event table telemetry and offset tokens.
What to remember for the exam
- Snowsight per task Run History covers 7 days;
TASK_HISTORYfunction for recent,ACCOUNT_USAGEfor long history. ERROR_INTEGRATIONon tasks supports SNS, Event Grid and Pub/Sub; email and webhook come through task alerts.- Stale streams must be recreated;
SYSTEM$STREAM_HAS_DATAchecks for changes. - Pipes:
SYSTEM$PIPE_STATUS,COPY_HISTORY,PIPE_USAGE_HISTORY. Streaming: event tables and offset tokens. - New tasks are suspended;
SUSPEND_TASK_AFTER_NUM_FAILURESstops repeated failures.
Check yourself
Which SQL function returns TRUE when a stream has pending changes?
SYSTEM$STREAM_HAS_DATA('stream_name').
What does a Snowpipe Streaming client use to resume safely after a failure?
The last committed offset token it stored for the channel.
Which integration types can a task ERROR_INTEGRATION use?
Cloud queue and topic services: Amazon SNS, Azure Event Grid and Google Pub/Sub, not email or webhook.
Topic 6 of 7 in Performance Optimization
Test yourself on Performance Optimization
The full Snowflake DEA-C02 question bank has 709 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Snowflake. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.