◀ Course contents Part 4 · Module 4-03

Root Cause Analysis

Fix the cause, not the symptom

Something broke, or a number fell off a cliff. Root cause analysis is the discipline of finding what actually produced it, instead of patching the nearest visible thing and waiting for it to come back. This guide covers the diagnostic sweep, the techniques, and how to prove a cause rather than agree on one.

Ready?

1

Symptom, Proximate Cause, Root Cause

Checkout conversion falls 12%. You could ship a fix this afternoon, and depending on which fix, it will either hold for a week or hold forever.

Three layers sit underneath any problem, and telling them apart is most of the skill.

The symptom is what you noticed — the 12%. It is a notification, not a problem. The proximate cause is what immediately produced it: the payment step is timing out. The root cause is what allowed that to be possible: there is no timeout budget on a third-party call, so any slow partner can stall the whole page.

Each layer has a real fix and they cost wildly different amounts. Put a banner on the page and the problem returns tomorrow. Add a retry and it returns more slowly, with a different partner. Only the third fix stops the clock — and only the third takes real engineering time, which is exactly why it gets deferred.

A puddle on the kitchen floor makes it concrete. Mopping is a legitimate action; it is just not a fix, because the dishwasher pipe is still leaking. Most roadmaps contain a surprising number of mops, and they are usually there because each one was individually cheap and urgent.

Symptom, proximate cause, root cause Three stacked layers: the symptom you can see, the proximate cause immediately behind it, and the root cause underneath that. Fixes applied at the top have to be reapplied. Symptom checkout conversion fell 12% Proximate cause the payment step times out Root cause no timeout budget on a third-party call deeper
All three rows are true statements about the same incident, and a fix aimed at any of them will make the number move. Only the keylined one makes it stay moved — and it is the only one that costs engineering time rather than an afternoon, which is the whole reason the top two get chosen.

Everyday example, the leaking ceiling

Symptom: water on the floor. Proximate cause: a crack in the ceiling. Root cause: a blocked gutter that's been overflowing into the wall cavity for months. You can mop the floor daily and even patch the ceiling, and the water will keep coming back, because neither action touches the gutter.

Symptom-fixing has a distinctive signature: the fix works, then has to be repeated, then has to be repeated bigger. More support agents for an overflowing queue. A second retry for a flaky call. A discount to hold a churning account. Each one buys time and none of them removes what generates the problem.

RCA also has a cost. Reserve the deep version for problems that recur, are expensive, or carry safety and trust risk. A 1% wobble on a secondary metric deserves a note and a watchlist, not a two-week investigation.

Three layers, three different fixes

Checkout conversion fell 12%. The symptom was the number. The proximate cause was a payment step timing out. The root cause was that a third-party call had no timeout budget, so one slow partner could stall the whole page. Fixing the symptom meant a banner; fixing the proximate cause meant a retry; only the third fix stopped it happening again with a different partner.

Quick check

A support queue keeps overflowing, so the team adds three agents. Six weeks later it overflows again. What did the team do?

2

The Diagnostic Sweep for a Metric Drop

A metric drops. Before theorising, run a sweep — a fixed order of cheap checks that rules out whole categories of explanation. It takes about twenty minutes and routinely saves a week.

Is it real? Check the tracking first, always. Analytics breakage produces a perfect, alarming cliff and no change in behaviour whatsoever. When exactly did it start? A sharp edge points at a release or an outage; a gradual slope points at a mix shift or a seasonal effect.

Who is it happening to? Segment by platform, channel, geography, plan, and new versus returning. A drop concentrated in one segment is a much smaller problem than the headline suggests, and a much easier one to explain.

What changed? Your releases, yes — but also a partner’s, an app store’s, a competitor’s pricing, a public holiday, a marketing campaign that ended.

The shapes carry information once you know them. A sharp drop with total silence from support is almost always instrumentation, because real breakage generates complaints within the hour. A slow slide across every segment is usually mix or market. A drop confined to one segment is usually a release.

Run the sweep even when you are confident — confidence is exactly the condition under which people skip step one and spend three days building an elegant explanation for a tracking bug.

1

Is it real?

Check the tracking first, always. Broken event, renamed property, SDK upgrade, failed pipeline job, changed metric definition — instrumentation produces a perfect, alarming cliff and no change in behaviour whatsoever.

2

When exactly did it start?

A sharp edge points at a release or an outage on a specific date. A gradual slope points at something compounding: a mix shift, decay, a seasonal effect.

3

Who is it happening to?

Segment by platform, app version, channel, geography, plan tier, new versus returning. A drop concentrated in one segment is a much smaller problem than the headline suggests, and a much easier one to explain.

4

What changed?

Your releases, yes — but also a partner's, an app store's, a competitor's pricing, a public holiday, a marketing campaign that ended.

The diagnostic sweep, in order Four checks in order of cost: is the drop real, when exactly did it start, who is it happening to, and what changed. Each rules out a whole category before any theorising begins. 1 Is it real? check tracking first 2 When exactly? sharp edge or slow slope 3 Who? segment before theorising 4 What changed? yours and everyone else’s
Twenty minutes, in this order, before anybody proposes a theory. Step one is the one that gets skipped, and it is skipped hardest when the team is certain — a tracking bug produces a perfect alarming cliff and complete silence from support, which is exactly what a confident explanation feels like from the inside.

Everyday example

A doctor with a patient in pain does not start with the rarest disease. They sweep: where does it hurt, when did it start, what changed, what else is different. A metric drop deserves the same order — segment, timing, instrumentation, external — before anyone theorises.

Signatures worth memorizing

A cliff-edge drop with silent support channels is almost always tracking. A drop confined to one country starting on a specific date, with no release, points outward: a competitor, a holiday, a payment provider, a regulator. A drop confined to one app version points straight at that release. A drop that's flat in every segment but visible in the aggregate is a mix shift, not a behavior change.

Segmentation doesn't find the cause. It shrinks the search space until the cause is findable.

The signature that named the cause

A metric fell off a cliff at 14:00 and support heard nothing at all. Silence alongside a sharp drop almost always means tracking rather than behaviour, and it did: a deploy had dropped an analytics call. The shape of the drop identified the cause before anyone opened a dashboard.

Quick check

Weekly active users drop 18% overnight. The team's first move is to brainstorm which product changes could explain it. What should come first?

3

Choosing the Right Technique

Five Whys is a chain. Ask why, take the answer, ask why again, five or so times. It works when the causation is genuinely linear: checkout crashes on Mondays → traffic spikes → a query gets slow → an index is missing → the migration that added it was skipped after a rollback → nobody checks migrations post-rollback. Four of those are fixable in an afternoon; the last one stops it recurring across every future feature.

Fishbone (or cause-and-effect) is a branching diagram. You list categories — people, process, technology, external — and gather contributing causes under each. It works when several things combine, which is the more common situation in product.

A quick way to choose: ask whether removing one cause would have prevented the problem entirely. If yes, the chain is real and Five Whys will find it. If the honest answer is “it would have helped a bit”, you have a multi-cause problem and you want the fishbone.

One chain

5 Whys

Ask why repeatedly down a single causal thread. Fast, cheap, and blind to parallel causes. End on a mechanism, never on a person.

Many causes

Fishbone (Ishikawa)

Fan one effect into categories, people, process, product, platform, policy, external, and enumerate contributors in each.

Search space

Issue tree

Break the problem into MECE branches you can test and eliminate. Best when the cause could be almost anywhere.

Prioritize

Pareto

Rank known causes by impact. Six error types out of forty often carry 80% of the failures.

A note on 5 Whys, since it's the one everyone reaches for. It fails in two predictable ways. It collapses multi-factor problems into one thread, so genuine contributors get dropped. And it terminates on people, "the developer didn't have enough time", which feels like an answer and isn't actionable. Push past it: why was the estimate wrong, why wasn't the risk visible, why did scope grow mid-sprint.

Which technique fits the problem A chooser. If removing one cause would have prevented the problem entirely, the causation is linear and Five Whys will find it; if several causes combine, a fishbone is the right tool. Would removing one cause have prevented it entirely? yes Five Whys — the chain is real, follow it down only partly Fishbone — several causes combine unsure Assume multi-cause. A wrong single answer is worse than none
One team ran Five Whys on a churn spike and arrived at “onboarding is too long” — one of four causes, and the smallest. The linear technique had forced a branching problem into a single line, and the quarter went to the branch that mattered least.

Everyday example

Five Whys works on a chain with one cause. A fishbone works when several causes combine. Using Five Whys on a multi-cause problem produces a confident single answer that is wrong, which is worse than no answer.

The wrong technique, confidently applied

A team ran Five Whys on a churn spike and arrived at "onboarding is too long". It was one of four contributing causes, and the smallest. The linear technique had forced a branching problem into a single line, and the team spent a quarter on the branch that mattered least.

Quick check

A problem has plausible contributors across engineering, design, pricing and support, and every 5 Whys attempt collapses into a single thread. Which technique fits better?

4

Proving the Cause, and Preventing the Recurrence

You have a plausible root cause. The temptation is to ship the fix and move on — and plausible is the exact point at which people stop being careful.

Proving it means finding evidence that would have looked different if you were wrong. If the cause is a timeout on a third-party call, the timings should be visible in the logs, the drop should be concentrated in the sessions that hit that call, and it should be absent from sessions that did not. A cause that explains the problem but predicts nothing else is a story, not a diagnosis.

Finally, prevention is a separate piece of work from repair, and it is the one that gets dropped when the incident is over and everyone is tired. If the root cause was “the migration was skipped after a rollback”, the fix is the index and the prevention is a check in the deploy process.

The question worth asking at the end of every investigation is not “is it fixed?” but “what would have to be true for this class of problem never to reach a customer again?” — and then deciding, deliberately and out loud, whether that is worth building today.

The prediction test

If this is the cause, what else must be true? "If latency drives abandonment, the slowest sessions should abandon most." Check it. A failed prediction kills a theory cheaply.

The recurrence test

Remove the cause and the effect stops. If the failure returns in a new costume two months later, you fixed a proximate cause.

Then make the fix structural. A countermeasure changes the system so the failure can't happen or is caught early: a pre-deploy check, a staged rollout, a guardrail, an alert that fires in minutes. "Be more careful" is not a countermeasure, it relies on vigilance that decays the week after the postmortem and can't be handed to a new joiner.

Proving a cause, and preventing it A checklist separating a proven diagnosis from a plausible story: the cause should predict evidence beyond the symptom, be checked against an unaffected segment, and be followed by a prevention step. Proven Plausible only Predicts evidence beyond the symptom Absent from sessions that avoided it Checked against a segment the fix missed Prevention built, not just repair Explains the symptom and nothing else "The metric recovered, so we were right" No control group at all Incident closed, process unchanged
One team shipped a fix and the metric recovered — along with three metrics the fix could not have touched, because something seasonal recovered too. A cause that explains the problem but predicts nothing else is a story, not a diagnosis.

Blameless, because you need the truth

A postmortem that hunts for who approved the change gets one name and no information. A blameless one assumes everyone acted reasonably given what they knew, and asks what in the system made the failure possible and hard to catch. The point isn't kindness, it's that people only report near-misses honestly in an environment where doing so is safe.

Proving it before believing it

A team identified a plausible root cause and shipped the fix. The metric recovered — and so did three other metrics that the fix could not have touched, which meant something seasonal had recovered too. Only by comparing against a segment the fix had not reached could they show any of the recovery was theirs.

Quick check

Two months after a root cause was "fixed", the same failure recurs in a slightly different form. What's the most likely explanation?

Drill what you learned
Scenario 1 easy

A support queue is overflowing, so a team adds three more support agents. Six weeks later the queue is overflowing again.

What did the team do?

Scenario 2 easy

A weekly active users number drops 18% overnight. The team's first move is to open a brainstorm on what product changes could have caused it.

What should come first?

Scenario 3 easy

Asked why checkout failed, an engineer answers "because the payment service timed out," and the team stops there.

What kind of cause is that?

Scenario 4 easy

A metric declines gradually over four months with no step change on any single day.

What does the shape of the decline suggest?

Scenario 5 easy

During a postmortem, most of the discussion focuses on which engineer approved the change that broke production.

What's wrong with this framing?

Scenario 6 medium

An app's crash rate spikes. Segmenting shows the spike is entirely on Android 14 devices running app version 5.2.

What has the segmentation accomplished?

Scenario 7 medium

A team runs 5 Whys on a delivery delay and lands on "the developer didn't have enough time," then stops.

What's the weakness in this chain?

Scenario 8 medium

A problem has many plausible contributing causes across engineering, design, pricing, and support, and 5 Whys keeps collapsing into a single thread.

Which technique fits better?

Scenario 9 medium

Analysis shows 6 error types account for 81% of all failed transactions, out of 40 distinct error types logged.

What principle does this illustrate, and what follows from it?

Scenario 10 medium

A team believes a slow API is causing checkout abandonment. Before committing engineering time, they want to test the belief.

What's the strongest check?

Scenario 11 hard

A revenue drop is traced to a pricing page change, but the same change shipped three months earlier in another market with no effect.

What does this tell the team?

Scenario 12 hard

An incident is closed with the corrective action "be more careful during deploys."

Why is this a weak countermeasure?

Scenario 13 hard

Two months after a root cause was "fixed," the same failure recurs in a slightly different form.

What's the most likely explanation?

Scenario 14 hard

A metric drop is investigated for two weeks with no conclusion, while the team's roadmap stalls. The drop is 1.5% on a secondary metric.

What's the judgement call here?

Scenario 15 hard

An engagement drop is confirmed real, isolated to one country, starting on a specific date, with no release that day.

What's the next best line of inquiry?

Drilled it. Now apply it to a real situation.

Put it to work
From lesson 1

Why the machine stopped

Toyota

The five whys comes from Toyota, where Taiichi Ohno used it to teach the difference between fixing a fault and fixing what produced it. His canonical example runs:

The machine stopped — a fuse blew. Why? The bearing seized. Why? It was not lubricated enough. Why? The lubrication pump was not pumping. Why? Its shaft was worn. Why? There was no filter, so metal filings got in.

Order these responses from the shallowest fix to the one that ends the problem.

Drag the rows, or use the arrows, then check.

  1. Replace the fuse Shallowest. Treats the symptom, and the machine stops again tomorrow. It is also the fastest and sometimes genuinely the right thing to do first — as long as nobody calls it the fix.
  2. Replace the bearing One level down. Addresses what blew the fuse, and it will seize again for exactly the same reason.
  3. Replace the pump shaft Closer to the mechanism, and still downstream of the actual cause. The new shaft wears out the same way.
  4. Fit a filter to the pump The root. It is the only step where a cheap change breaks the whole chain — and it prevents the failure rather than repairing it.
From lesson 2

Signups fell 30% overnight

A fitness app

Tuesday morning, signups down 30% against a stable four-week baseline. Nothing was deployed overnight. Six people have six theories and the loudest one is "the ad agency must have paused a campaign".

Order the diagnostic sweep, first move to last.

Drag the rows, or use the arrows, then check.

  1. Confirm the drop is real — check the tracking is firing and the dashboard is not broken First, always. A meaningful share of "sudden drops" are a broken event, an expired tag, or a dashboard filter someone saved. Ten minutes here saves a day of chasing a number that never fell.
  2. Segment it — device, browser, channel, country, new versus returning Second. This is what converts "signups fell" into "signups fell on Android in the UK", which eliminates most theories without testing any of them.
  3. Line up the timeline — releases, config changes, third-party incidents, campaign schedules Third, and only once you know which segment. A change log is far more useful when you already know you are looking for something that could affect Android in one country.
  4. Form a specific hypothesis and test it against the data Last. A hypothesis raised before the first three steps is a guess competing with five other guesses; raised after them it is usually the only one still standing.
From lesson 3

Five whys was the wrong tool

A logistics platform

Deliveries are late. A five whys session produces: late because drivers leave the depot late, because loading takes too long, because pick lists are wrong, because stock locations are stale, because nobody updates them.

Fix the stale locations and lateness improves by 6%. It was expected to solve it.

Work out why the technique underperformed.

  1. Step 1 of 3

    What does five whys assume that may not hold here?

    The 6% is the size of that one branch. The rest of the lateness lives on branches nobody walked.

  2. Step 2 of 3

    Which technique fits a problem with several parallel causes?

    The fishbone surfaced four categories, of which loading was one. Route planning turned out to be larger.

  3. Step 3 of 3

    When would five whys have been the right choice?

    One symptom and one chain: five whys. One symptom and many contributors: fishbone. Many symptoms: segment first.

From lesson 4

Proving it, then preventing it

A ticketing platform

Investigation concludes that checkout failures spike because a third-party fraud check times out under load. It is a plausible chain and everyone believes it. The team is about to spend a quarter replacing the vendor.

What would actually establish that this is the cause, and stop it recurring?

Select all that apply — there are 3 to find.

Notification