Monitor and Optimize an Analytics Solution · Topic 2 of 5
Resolving Pipeline, Dataflow Gen2 and Notebook Errors
DP-700 troubleshooting: fix pipeline activity errors with retries and failure paths, Dataflow Gen2 refresh and destination errors, and notebook failures.
- 6 min
- Pipeline errors are in each activity's error and output; the failing activity is marked in the run.
- Retry, retry interval and timeout make activities resilient to transient failures.
- Dataflow Gen2 refresh history drills into failing tables and activities.
What it is
Batch processes fail for a small set of reasons: connections and credentials, permissions, schema or data type changes, resource limits and code bugs. Resolving an error means finding the failing step, reading its message, and fixing the cause rather than rerunning blindly.
Each item shows errors in its own place: activity output for pipelines, refresh history for Dataflow Gen2, and cell output and Spark logs for notebooks.
How it works
Pipeline errors
Open the failed run from the Monitor hub or the pipeline's run history. The activity list marks the failing activity; its error and output JSON hold the message and error code.
| Symptom | Likely cause | Fix |
|---|---|---|
| Copy fails with an authentication error | Expired password, key or token in the connection | Update the connection's credentials |
| Source unreachable | On-premises data gateway offline, or the source is down | Restart or update the gateway, check the source |
| Copy fails on a column | Source schema changed, or a type cannot convert | Fix the mapping or the destination table |
| Expression error | Wrong path such as a misspelled activity name, or a null value | Correct the expression; use coalesce() for possible nulls |
| Notebook activity fails | An exception inside the notebook | Open the notebook snapshot for that run |
Build pipelines to handle failure:
- Set retry and retry interval on activities that call flaky sources, and a timeout so a hung activity does not run for hours.
- Connect an on failure path to a notification activity or a logging step.
- Use a Fail activity to stop the run with a clear message and error code when a check fails, such as zero rows loaded.
A failure path that succeeds can make the whole run report success, which hides the failure from monitoring and alerts. If the run must still show as failed, end the failure path with a Fail activity.
Dataflow Gen2 errors
Open Refresh history, select the failed refresh, and drill into the table and activity that failed.
| Symptom | Likely cause | Fix |
|---|---|---|
| Credentials error | The connection's sign-in has expired | Edit the connection and sign in again |
| Data destination error | Destination column types or names differ from the query output | Align types, or let the destination use automatic settings that adapt the schema |
| Conversion errors in a column | A value cannot change to the set type | Fix the step, or use Replace errors, Remove errors or Keep errors to inspect |
| Refresh slow or times out | Large data without query folding, or heavy steps | Push filters to the source so they fold, use staging, or move heavy logic to a notebook |
| Gateway errors | Gateway offline or out of date | Restart or update the on-premises data gateway |
Dataflow Gen2 can stage queries in a lakehouse and warehouse managed by Fabric. Staging speeds up large transformations, but adds a write step; turn it off for small queries that do not need it.
Notebook errors
The failing cell shows the exception, and the Spark application details show jobs, stages, executor logs and the Spark UI.
| Error | Likely cause | Fix |
|---|---|---|
AnalysisException such as table or column not found | Wrong name, or no default lakehouse attached | Correct the name, attach the default lakehouse, or use a full path |
ModuleNotFoundError | Library missing from the session | Add it to the attached environment |
| Out of memory on the driver | collect() or toPandas() on a large DataFrame | Keep data distributed; write results to a table |
| Out of memory on executors, or very slow stages | Data skew or nodes too small | Repartition, or use a larger node size |
| Session or job not started, HTTP 430 | Capacity Spark limits reached; jobs are queued or rejected | Wait, reduce concurrent sessions, or scale the capacity |
Relative paths such as Tables/sales and unqualified table names resolve against the notebook's default lakehouse. A notebook that works for one user can fail in a pipeline if it is deployed to another workspace with a different default lakehouse.
Wrap risky steps in try/except, log the error, and re-raise so the pipeline's Notebook activity still sees the failure.
try:
df = spark.read.table("silver_orders")
df.write.mode("append").saveAsTable("gold_orders")
except Exception as e:
print(f"Load failed: {e}")
raiseA worked example
A pipeline that ran for months fails at the Copy activity with a type conversion error on Discount. The source team changed Discount from an integer to a decimal.
The engineer reads the activity's error output, confirms the column in the source, and changes the destination column to a decimal type in the mapping and the table. They also add a Fail activity after a row count check, so an empty load stops the pipeline with a clear message instead of passing silently.
How the exam asks about this
- "A Copy activity fails intermittently because the source times out." Configure retry and retry interval on the activity.
- "Stop the pipeline with a custom error when a validation fails." The Fail activity.
- "A notebook fails with a table not found error when run from a pipeline." Check the notebook's default lakehouse.
What to remember for the exam
- Pipeline errors are in each activity's error and output; the failing activity is marked in the run.
- Retry, retry interval and timeout make activities resilient to transient failures.
- Dataflow Gen2 refresh history drills into failing tables and activities.
- Destination type mismatches and expired credentials are the most common Dataflow Gen2 failures.
- Notebook relative paths depend on the default lakehouse;
collect()on big data exhausts the driver.
Catching an exception in a notebook without re-raising it makes the Notebook activity succeed. The pipeline then continues as if the load worked.
Common mistakes
Rerunning a failed pipeline without reading the error. Credential and schema errors fail again; fix the cause first.
Fixing a type error by converting everything to text. It hides the problem and breaks downstream calculations; set the correct type.
Raising node size for a skewed job. Larger nodes help a little; repartitioning or salting the skewed key fixes the cause.
Check yourself
Where do you find the error message for a failed pipeline activity?
In the pipeline run details, in the failed activity's error and output.
What should you check first when a Dataflow Gen2 refresh fails writing to its destination?
Whether the query output's column names and types match the destination table, or the destination's schema settings.
Why can a notebook find a table interactively but fail in another workspace?
Unqualified table names and relative paths use the default lakehouse, which can differ or be missing in the other workspace.
Topic 2 of 5 in Monitor and Optimize an Analytics Solution
Test yourself on Monitor and Optimize an Analytics Solution
The full Microsoft DP-700 question bank has 645 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Microsoft. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.