Pipelines: testing strategy · Topic 2 of 2
Testing Strategy, Test Tasks and Code Coverage
AZ-400 testing: the test pyramid, Azure Load Testing, test tasks and agents in pipelines, publishing results, flaky tests and code coverage thresholds.
- 5 min
- Test pyramid: many unit tests, fewer integration tests, few UI tests. Load tests check performance.
PublishTestResults@2publishes results. UsesucceededOrFailed()so failures are recorded.- Azure Load Testing uses JMeter scripts and fail criteria to gate a pipeline.
What it is
A testing strategy decides which kinds of tests exist, who writes them, where they run and what happens when they fail. A comprehensive strategy mixes fast, cheap tests that run on every commit with slower, broader tests that run less often.
This topic covers the test types, how to run tests and publish the results in a pipeline (test tasks and agents), and code coverage analysis, which measures how much of the code the tests execute.
How it works
The test pyramid
Many cheap tests at the bottom, few expensive ones at the top.
| Level | Test type | What it checks | Speed |
|---|---|---|---|
| Base | Unit tests | One function or class in isolation, with dependencies replaced by fakes | Milliseconds |
| Middle | Integration tests | Components together, such as code plus a database or an API | Seconds |
| Upper | End to end / UI tests | The whole system through its interface | Minutes |
| Specialized | Load and performance tests | Behavior under traffic: response time, throughput, errors | Minutes to hours |
| Specialized | Security tests | Static analysis, dependency, dynamic scans | Varies |
Other terms the exam uses: smoke tests (a quick check that the deployment basically works), regression tests (existing behavior still works), acceptance tests (the feature meets the requirement) and local tests that developers run on their own machine, in the IDE or in a pre-commit step, before pushing.
Load testing
Azure Load Testing runs high scale load tests against an app. You provide a JMeter script (or a URL based test), and the service generates traffic from managed engines. In a pipeline you add the Azure Load Testing task or action, point it at a test, and define fail criteria, for example failing the run when the 95th percentile response time exceeds a limit or the error rate is above a percent. The pipeline then fails automatically when the app is too slow.
Running tests in a pipeline
steps:
- task: DotNetCoreCLI@2
inputs:
command: test
projects: '**/*Tests.csproj'
arguments: '--configuration Release --collect:"XPlat Code Coverage"'
- task: PublishTestResults@2
condition: succeededOrFailed()
inputs:
testResultsFormat: JUnit
testResultsFiles: '**/TEST-*.xml'| Element | Notes |
|---|---|
| Test tasks | VSTest@3 runs .NET tests. Other runners (Maven, Gradle, npm scripts, pytest) are run by their own tasks or scripts |
| PublishTestResults@2 | Publishes results in formats such as JUnit, NUnit, xUnit, VSTest and CTest so they show on the run's Tests tab. Use condition: succeededOrFailed() so results publish even when tests fail |
| Test impact analysis | The VSTest task can run only tests affected by the change, to save time |
| Parallelism | Split tests across jobs or use a matrix to test several operating systems or versions at once |
| Test agents | Choose agents that match the tests: the right OS, installed browsers and tools. UI tests on a self hosted Windows agent need it to run interactively, not as a service |
| Flaky tests | Tests that sometimes pass and sometimes fail. Azure DevOps can detect and mark flaky tests so they do not block the build, but they should be fixed |
| Environments | Integration tests need a database or service. Use service containers, test deployments or ephemeral environments, and clean up afterwards |
Integrate results with work items: failed tests can create bugs, and Azure Test Plans links manual and automated tests to requirements for traceability.
Code coverage
Code coverage is the percentage of code executed while the tests run.
| Measure | Meaning |
|---|---|
| Line coverage | Share of lines executed |
| Branch coverage | Share of decision outcomes (both sides of each if) executed |
Collect and publish coverage like this.
| Platform | How |
|---|---|
| .NET | --collect:"XPlat Code Coverage" (Coverlet) produces Cobertura XML. Coverlet can fail the build below a threshold (/p:Threshold=80) |
| Java | JaCoCo reports, with rules that fail the build |
| Python | pytest --cov with --cov-fail-under=80 |
| Azure Pipelines | PublishCodeCoverageResults@2 shows the report on the run |
| GitHub | Upload the report to a coverage service or publish it as an artifact, and fail the job below the threshold |
Treat coverage as a floor, not a goal. A high number says the code ran, not that the assertions are good. Many teams enforce coverage on new or changed code so legacy code does not block progress.
A worked scenario
A team's pipeline takes 50 minutes. They keep unit tests with coverage on every pull request, move the integration tests to a later stage that runs after merge, split UI tests across parallel jobs, and add an Azure Load Testing step with fail criteria before staging. They publish results with PublishTestResults@2 and a coverage threshold of 80 percent for new code.
What to remember for the exam
- Test pyramid: many unit tests, fewer integration tests, few UI tests. Load tests check performance.
PublishTestResults@2publishes results. UsesucceededOrFailed()so failures are recorded.- Azure Load Testing uses JMeter scripts and fail criteria to gate a pipeline.
- Coverage: Coverlet (.NET), JaCoCo (Java), pytest-cov (Python). Publish with
PublishCodeCoverageResults@2. - Flaky tests are detected and managed, not ignored. Test impact analysis runs only affected tests.
100 percent coverage does not mean the code is correct. Coverage shows what ran, not what was verified.
How the exam asks about this
- "Fail the pipeline if response time is too slow." Azure Load Testing with fail criteria.
- "Show test results even when a test fails." Publish with a condition of succeeded or failed.
- "Run tests on Windows and Linux." A matrix strategy across agent pools.
- "Reduce test time by running only affected tests." Test impact analysis.
Common mistakes
Only UI tests. They are slow and brittle. Build on unit tests.
Not publishing results. Without it, failures are buried in logs.
Chasing a coverage number. It leads to tests with no assertions.
Ignoring flaky tests. They teach the team to distrust the pipeline.
Check yourself
Which test type should make up the largest share of a strategy?
Unit tests.
Which task publishes test results to an Azure Pipelines run?
PublishTestResults@2.
What does Azure Load Testing use to fail a pipeline when the app is too slow?
Fail criteria defined on the test, such as a response time or error rate limit.
Topic 2 of 2 in Pipelines: testing strategy
Test yourself on Pipelines: testing strategy
The full Microsoft AZ-400 question bank has 766 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Microsoft. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.