Pipeline Health and Optimization

What it is

Pipeline health is how reliably and quickly your pipelines do their job. A healthy pipeline passes when the code is good, fails when it is bad, finishes fast and costs little. Optimization is the work of moving a pipeline toward that: less waiting, fewer false failures, lower bills.

Both start with measurement. Without numbers, people optimize what they can see rather than what matters.

How it works

What to monitor

Where to see it

Set alerts on the trends that matter: a spike in failures on main, or a run that takes twice its normal time.

Flaky tests

A flaky test gives different results with no code change, because of timing, shared state, test order or an unreliable dependency.

• Detect: Azure Pipelines test analytics can identify flaky tests, and it can rerun failed tests (the VSTest task supports rerunning failures) to separate real failures from noise.
• Contain: mark known flaky tests so they do not block the build, but keep them visible.
• Fix: remove shared state, replace sleeps with waits for conditions, and isolate external calls. Track a flaky test as a bug with an owner.

Optimizing time

- task: Cache@2
  inputs:
    key: 'npm | "$(Agent.OS)" | package-lock.json'
    restoreKeys: |
      npm | "$(Agent.OS)"
    path: $(Pipeline.Workspace)/.npm

Optimizing cost

• Cancel redundant runs. In Azure Pipelines, pull request validation runs are cancelled when a newer commit arrives. In GitHub use concurrency with cancel-in-progress: true.
• Avoid needless runs with path filters, [skip ci] and batching.
• Right size the agents. Do not use a large machine for a lint job.
• Scale self-hosted capacity to zero when idle and use cheaper capacity for non urgent work.
• Clean up artifacts and old runs so storage does not keep growing (see retention).

Optimizing concurrency

The number of jobs that can run at once is limited by licensed parallel jobs in Azure Pipelines, by plan limits in GitHub, and by your own agent count. If queue time is high, add capacity or reduce wasted runs. If queue time is near zero, more parallelism only adds cost. Use maxParallel on matrices and exclusive locks where concurrent deployments to one target would conflict.

Optimizing reliability

• Pin tool and action versions so an upstream release does not break the build.
• Get dependencies from your own feed with an upstream source rather than directly from public registries.
• Use retryCountOnTaskFailure for steps that depend on the network, and timeoutInMinutes so a hung job cannot hold an agent.
• Keep scripts idempotent and use ephemeral agents so a job cannot be broken by leftovers from earlier runs.
• Keep agent software and images patched.

A worked scenario

A team's main pipeline takes 40 minutes and fails once in five runs. Analytics show 15 minutes in package restore and that three tests produce most failures. They add caching keyed on the lock file, fetch with fetchDepth: 1, split the test suite across four parallel jobs and quarantine the three flaky tests with tickets. They also cancel superseded pull request runs. Duration falls to 14 minutes and the failure rate drops.