Automated Testing in Pipelines

What it is

Automated testing in a pipeline means every change is checked by code, not by a person, before it can reach production. Tests run at several points: before the build finishes, after a deploy to a test environment, and after the release to production.

The aim is fast feedback and a safe gate. A failed test stops the pipeline, so a bad change never reaches the next stage.

How it works

The test layers

Put cheap, fast tests first and expensive ones later. This is the idea behind the test pyramid.

CodeBuild as the test runner

In CodePipeline a CodeBuild action can sit in the Build category or the Test category. If a command returns a non zero exit code the build fails, and the stage and the pipeline stop. Test result files are published as reports, so you can see trends even when the build passes.

version: 0.2
phases:
  build:
    commands:
      - pytest --junitxml=report.xml
reports:
  pytest-results:
    files:
      - report.xml
    file-format: JUNITXML

Running tests on pull requests and merges

Validate a change before it merges, not only after. Three common triggers exist.

aws codebuild create-webhook --project-name app-tests \
  --filter-groups '[[{"type":"EVENT","pattern":"PULL_REQUEST_CREATED, PULL_REQUEST_UPDATED"},{"type":"BASE_REF","pattern":"^refs/heads/main$"}]]'

Report the result back to the pull request so reviewers see it. With a webhook, CodeBuild posts the build status to the source provider. With CodeCommit, a Lambda function or the build itself can add a comment to the pull request or update its approval state.

Testing after a deploy

An integration test needs a running environment. The usual pattern is to deploy to a test stack, run a CodeBuild action against its endpoint, and then tear the stack down or keep it for the next run.

A Lambda function in an Invoke action suits small custom checks, such as calling a health URL and reporting success or failure back to the pipeline.

Testing inside the deployment

CodeDeploy can run tests as part of the deploy itself.

• On EC2, the ValidateService hook runs a script after the application starts and can fail the deployment.
• For Lambda and ECS, the traffic hooks (BeforeAllowTraffic and AfterAllowTraffic, plus AfterAllowTestTraffic for ECS) call a Lambda function that tests the new version.
• CodeDeploy can watch CloudWatch alarms during a deployment and stop and roll back when one enters the ALARM state.

This turns production monitoring into the last test: a canary release sends a small share of traffic to the new version while alarms judge its health.

Security testing

Shift security checks left, which means running them early in the pipeline.

• Scan source and dependencies in the build for known vulnerable packages and committed secrets.
• Scan container images in Amazon ECR, using either the basic scan or Amazon Inspector enhanced scanning.
• Validate infrastructure templates before deploy with tools such as cfn-lint and CloudFormation Guard.
• Use AWS CloudFormation Hooks to block noncompliant resources at deploy time.

Gates and approvals

A gate can be automatic (tests must pass, alarms must be clear) or manual (an approval action). Use automatic gates for repeatable rules and manual approval where a business decision is needed. Many teams use a manual approval only for the production step.

Test data and isolation

Tests should not depend on shared long lived state. Give each pipeline run its own test stack or its own data prefix, and use short lived credentials from the pipeline role.