Pipelines: design and implement deployments · Topic 2 of 3
Hotfix Paths, Resiliency and Feature Flags
AZ-400 safe delivery: hotfix paths, rollback versus roll forward, resilient deployments, and feature flags with Azure App Configuration Feature Manager.
- 6 min
- Hotfix: branch from the production tag, run the same pipeline, then merge back to main.
- Roll back when the old version still works with the data. Roll forward after one way changes.
- Safe multi-region rollout: one region at a time with bake time.
What it is
Even a good pipeline will sometimes ship a bad change. This topic is about being ready for that: a hotfix path for urgent fixes, a resiliency strategy so a failed deployment does not become an outage, and feature flags that let you switch behavior on or off without a new deployment.
How it works
A hotfix path plan
A hotfix is a high priority fix to production code that cannot wait for the next normal release.
- Branch from what is in production: the release tag or release branch, not the current
main, which may contain unreleased work. - Make the smallest fix and a test that proves it.
- Run the same pipeline with a possibly reduced but still meaningful test set. Do not hand deploy from a laptop.
- Approve quickly: a named on call approver, and an agreed expedited path. An administrator may bypass a non critical check, and the bypass is recorded.
- Deploy, verify with health checks, and tag the release.
- Merge or cherry pick the fix back into
main(and other supported release branches) so the bug does not return in the next release.
Document the path, practise it, and keep it short. A hotfix path nobody has tried will fail when needed. With trunk based development and feature flags, a hotfix is often just a normal small change, or a flag turned off.
Rollback or roll forward
| Option | When to use |
|---|---|
| Roll back (redeploy the previous artifact, swap the slot back, switch blue-green) | The previous version still works with current data and the fix is not quick |
| Roll forward (deploy a fix) | A rollback is unsafe, for example after a one way data migration, and a fix is fast |
| Turn off a feature flag | The problem is in a flagged feature. The fastest option |
Rollback needs the previous artifact kept and versioned, and database changes that are backward compatible.
A resiliency strategy for deployment
Resilience here means a deployment can fail safely.
| Practice | Effect |
|---|---|
| Automated health verification | After deploy, run smoke tests and check metrics. Fail the stage if they fail |
| Automatic rollback | A failure hook swaps the slot back or redeploys the last good version |
| Gradual exposure | Canary, rings and rolling updates limit the damage |
| Safe multi-region rollout | Deploy to one region, wait a bake time while monitoring, then the next. Never update all regions at once |
| Immutable, versioned artifacts | The same artifact deploys everywhere, so you can redeploy exactly what worked |
| Idempotent deployment scripts | Running twice gives the same result, so retries are safe |
| Retries and timeouts | retryCountOnTaskFailure retries a flaky step. timeoutInMinutes stops a hung one |
| Backups before change | A restore point for data before risky migrations |
| Alerts and monitoring tied to the pipeline | A Query Azure Monitor alerts check stops promotion when the app is unhealthy |
Feature flags
A feature flag (feature toggle) is a condition in code that turns functionality on or off at run time. It separates deployment (code is in production) from release (users can use it).
Benefits: ship incomplete work safely behind a flag, release to a few users first, turn a feature off instantly (a kill switch) and run experiments.
Azure App Configuration Feature Manager
Azure App Configuration is a managed store for settings and feature flags. Its Feature manager is where you create and control flags.
| Element | Notes |
|---|---|
| Feature flag | A name and an enabled or disabled state, stored in the App Configuration store |
| Feature filters | Rules for when an enabled flag is on. Built in filters are Percentage (a share of requests), Time window (start and end), and Targeting (specific users or groups with a rollout percentage) |
| Labels | Different values for different environments, such as Production and Test |
| Feature management libraries | Packages for .NET, Java (Spring), JavaScript and Python that read flags and evaluate filters |
| Refresh | The application can refresh flags at run time, so a change takes effect without a restart |
A .NET example:
if (await featureManager.IsEnabledAsync("NewCheckout"))
{
return View("CheckoutV2");
}
return View("Checkout");You can also manage flags from automation with the Azure CLI (az appconfig feature set, enable and disable), so a pipeline can turn a flag on after a successful deployment. Use managed identity for the application to read the store, not connection strings in code.
Flags need housekeeping. Remove flags once a feature is fully released, or the code fills with dead branches, and record who owns each flag.
A worked scenario
A team ships a new checkout behind a flag called NewCheckout. They deploy on Tuesday with the flag off. On Wednesday they enable a Targeting filter for employees, then a Percentage filter at 10 percent, watching error rates. When the new checkout causes timeouts, they disable the flag in Feature manager within a minute, with no deployment. The fix goes through the normal pipeline, and the flag is deleted a month after full release.
What to remember for the exam
- Hotfix: branch from the production tag, run the same pipeline, then merge back to main.
- Roll back when the old version still works with the data. Roll forward after one way changes.
- Safe multi-region rollout: one region at a time with bake time.
- Azure App Configuration Feature manager: flags, built in filters (Percentage, Time window, Targeting), labels, libraries.
- Feature flags separate deployment from release and give a kill switch.
Never branch a hotfix from main if main holds unreleased work. You would ship that work as well.
How the exam asks about this
- "Turn off a problematic feature instantly without redeploying." A feature flag in Azure App Configuration.
- "Release a feature to 10 percent of users." The Percentage feature filter, or Targeting with a rollout.
- "Where does a hotfix branch start?" From the release tag or release branch running in production.
- "Avoid taking down every region with one bad release." Deploy one region at a time with a bake period.
Common mistakes
Hand fixing production. Skipping the pipeline loses traceability and tests.
Forgetting to merge the hotfix back. The next release brings the bug back.
Leaving old flags in code. Remove them after release.
Rollback without compatible data. Plan schema changes so the previous version still runs.
Check yourself
Which built in feature filter turns a flag on for a share of requests?
Percentage.
Where should a hotfix branch be created from?
The release tag or branch that is running in production.
What makes a deployment script safe to retry?
Being idempotent, so running it again gives the same result.
Topic 2 of 3 in Pipelines: design and implement deployments
Test yourself on Pipelines: design and implement deployments
The full Microsoft AZ-400 question bank has 766 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Microsoft. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.