Change failure rate: the DORA metric everyone measures wrong

The metric that flatters you until it doesn't

Of the four DORA metrics, change failure rate is the one leaders quote most confidently and understand least. Deploy frequency and cycle time are at least mechanical — you count deploys, you measure the clock. Change failure rate asks a harder question: of the changes you shipped, how many broke something? And "broke something" is where every team quietly invents its own definition, usually the one that makes the dashboard look good.

I've watched a platform org report a 3% change failure rate to the board while the on-call channel was a nonstop stream of rollbacks. The number wasn't a lie exactly. It was just measuring a category of failure so narrow that most of the actual pain fell outside it. That's the problem with this metric. It's easy to compute and easy to game, often by accident.

So let's do it properly. What change failure rate is, how to measure it without fooling yourself, what it genuinely tells you about engineering productivity, and the specific ways teams turn it into a vanity number.

What change failure rate actually means

The DORA definition: the percentage of deployments to production that result in a degraded service and require remediation — a hotfix, a rollback, a patch, a forward-fix. The formula is boring:

change failure rate = failed deployments / total deployments

The elite-performer band in the DORA research sits somewhere between 0 and 15%. That range is wide on purpose, because the denominator and the numerator both depend on decisions you haven't made yet.

The interesting fights are all in the definitions. What counts as a "deployment"? What counts as a "failure"? Get those two wrong and the ratio tells you nothing.

Measuring it without lying to yourself

Start with the denominator, because it's the easier half. A deployment is a distinct release of code to production. If you deploy 40 times a day with continuous deployment, that's 40 events. If you ship a monolith once a week behind a big-bang release, that's one. Already you can see the trap: two teams with identical reliability will report wildly different change failure rates purely because of batch size. The weekly team's single failure is 100% of that day's deploys. The continuous team's single failure is 2.5%.

This is why change failure rate should never be read alone. Pair it with deploy frequency and cycle time, because the three constrain each other. A gorgeous change failure rate next to a deploy frequency of "twice a month" isn't excellence — it's fear.

Now the numerator, which is where honest measurement lives or dies. You need a consistent, machine-detectable signal for "this deployment failed." The cleanest sources:

• Rollbacks and reverts. A deploy immediately followed by a rollback or a git revert of the release commit.
• Hotfix deploys. A deploy tagged or labeled as a fix for a recent release, ideally within a time window (say, 24 hours).
• Incidents linked to a deploy. If your incident tooling captures the triggering change, that's your strongest signal.

Here's a rough shape for deriving it from deploy events, assuming each deploy carries metadata:

def change_failure_rate(deploys, incidents, window_hours=24):
    failed = 0
    for d in deploys:
        if d.rolled_back:
            failed += 1
            continue
        # a hotfix or incident tied to this deploy within the window
        linked = [
            i for i in incidents
            if i.caused_by_deploy == d.id
            and (i.opened_at - d.finished_at).total_seconds() <= window_hours * 3600
        ]
        if linked:
            failed += 1
    return failed / len(deploys) if deploys else 0.0

Notice what this code forces you to do: attribute failures to deploys. That's the hard, valuable work. If you can't connect an incident back to the change that caused it, you don't have a change failure rate — you have an incident count wearing a costume.

Measure it as a trailing rate — a rolling 30-day window per service, plus an org roll-up. Point-in-time percentages are noise, especially for teams that deploy infrequently. You want the slope, not the snapshot.

What it tells you — and what it absolutely does not

Change failure rate is a quality-of-change signal. Rising numbers tell you the org is shipping changes that don't survive contact with production. That's genuinely useful. It's one of the few metrics that pushes back against the "just ship faster" reflex, because it makes the cost of recklessness visible.

What it does not tell you:

• Severity. A one-line copy fix that flips a feature flag counts the same as an outage that took payments down for an hour. The metric treats a paper cut and an amputation as one failure each. If you want to reason about impact, you need severity-weighted variants or you need to read it alongside MTTR.
• Where the failure came from. A high rate could mean bad code, thin test coverage, a fragile deploy pipeline, or a review process that waves everything through. The number flags the symptom and stays silent on the cause.
• Whether the failure was caught by design. Teams with mature progressive delivery intentionally ship to a canary, watch it fail, and roll back automatically. That's the system working. Counting every auto-rollback as a failure punishes exactly the practice you want to encourage.

That last one matters more than it looks. The healthiest engineering orgs I've seen deliberately move failure detection earlier — into canaries, into staging, into CI. A crude change failure rate can make a team that catches problems well look worse than a team that ships blind and gets lucky. Read the metric with that context or it will steer you backward.

The ways teams misuse it

Turning it into a target. The moment change failure rate becomes a number an individual team is graded on, the definition of "failure" starts drifting. Rollbacks get reclassified as "planned config changes." Hotfixes get bundled into the next scheduled release so they don't register as a distinct fix. You didn't improve quality — you improved the paperwork. Goodhart's law comes for every metric you incentivize, and this one is especially soft.

Ignoring batch size. I said it above and I'll say it again because it's the most common error. Comparing change failure rate across teams with different deploy frequencies is comparing nothing. Normalize the conversation around deploy frequency first, or compare each team only against its own trend.

Confusing CI failures with change failures. This is the one that hits compliance-heavy shops hardest, and it's where the metric quietly rots. If a deploy is gated on a test suite and the suite fails intermittently, some teams count that blocked or reverted deploy as a change failure. It wasn't. The change was fine. The test was flaky. Now your change failure rate is inflated by CI noise that has nothing to do with the quality of your code.

The reverse is worse. When flaky tests train a team to reflexively rerun red builds, real regressions slip through the reruns and land in production — where they finally show up as genuine change failures weeks later, disconnected from the change that caused them. Flaky CI corrupts the metric from both directions: it invents failures that aren't real and it hides the ones that are. If you want change failure rate to mean anything, your CI signal has to be trustworthy first. That's not a tooling footnote — it's a precondition. We've written before about how reruns launder real bugs into green checks and why a flaky suite is a change-management risk, not just an annoyance.

Measuring it manually. If a human decides what counts as a failure after the fact, you have a metric with a bias baked into every data point. The person filling in the spreadsheet knows what number makes their team look good. Derive it from deploy events, rollbacks, and incident links — automatically — or don't bother reporting it upward.

Wiring it up in CI

The raw material is deploy metadata plus rollback and incident signals. If your deploys run through GitHub Actions, tag every production release so you can attribute failures later:

- name: Record deploy
  run: |
    curl -sf -X POST "$METRICS_URL/deploys" \
      -H "Authorization: Bearer $METRICS_TOKEN" \
      -d service=checkout \
      -d sha="${GITHUB_SHA}" \
      -d env=production \
      -d actor="${GITHUB_ACTOR}" \
      -d run_id="${GITHUB_RUN_ID}"

And on rollback, close the loop by marking the original deploy as failed:

- name: Record rollback
  if: ${{ github.event.inputs.rollback == 'true' }}
  run: |
    curl -sf -X POST "$METRICS_URL/deploys/${TARGET_SHA}/fail" \
      -H "Authorization: Bearer $METRICS_TOKEN" \
      -d reason=rollback

That's enough to compute an honest trailing rate. The judgment — severity weighting, excluding intentional canary rollbacks, connecting incidents back to deploys — is the part you can't skip and can't fully automate. Which is fine. The metric is a starting point for a conversation, not a verdict.

Read it as one line in a story

Change failure rate earns its place when you read it next to its siblings. Deploy frequency tells you cadence. Cycle time tells you how fast an idea becomes production code. Change failure rate tells you how much of that speed you're paying back in rework. Mean time to restore tells you how badly it hurts when it goes wrong. No single one of those is engineering productivity. Together they sketch it.

The teams that get value from DORA metrics treat them as a diagnostic dashboard, not a scoreboard. They ask why the change failure rate moved, and the answer is almost never "the engineers got worse." It's usually a thinning test suite, a batch size that crept up, a deploy pipeline nobody trusts, or a CI signal so noisy that nobody can tell a real failure from a flake anymore.

Fix the signal first. Then the number starts telling the truth.