◀ Course contents Part 3 · Module 3-08

Analysis

Turn data into a decision, not just a chart

Data doesn't speak for itself, someone has to ask it the right question. This guide covers how to approach analysis with a complete mindset, which method fits which question, a few frameworks worth knowing, and the biases that quietly wreck good analysis.

Ready?

1

From Data to Decision

This is the difference between reporting and analysis. Reporting describes what happened. Analysis exists to change what someone does next. If no decision hangs on the answer, you are producing a document, and a document is a perfectly reasonable thing to produce as long as nobody mistakes it for thinking.

The practical consequence is that the sequence runs backwards from what feels natural. Start with the decision — what will we do differently depending on the answer? Then work out what evidence would move that decision, then go and get it. Most analysis starts with available data and hunts for something interesting in it, which is how you end up with forty tiles.

A useful discipline before opening any tool: write down the sentence you expect to be able to say at the end. “We should invest in onboarding rather than acquisition, because…” If you cannot draft that sentence, you do not yet know what you are looking for — and an analysis without a decision attached will always find something, which is precisely the problem.

Analysis runs backwards from the decision Four steps in reverse of the instinctive order: name the decision, identify what evidence would move it, gather only that, then answer. Starting from available data instead produces reporting. 1 Name the decision what changes either 2 Identify evidence what would move it 3 Gather that evidence, and only that 4 Answer in one sentence
Run this drawing right to left and it is what most analysis actually does: open the tool, find something, work out afterwards what it might mean. Both directions end in a document — only one of them knew, before it started, which document it was going to be.

Everyday example, how a doctor works

A doctor doesn't run every possible test on every patient. They form a hypothesis about what might be wrong, then order the specific tests that would confirm or rule it out. Good analysis works the same way: start with a question, then go find the data that answers it.

The dashboard nobody acted on

A team built a forty-tile dashboard covering every metric the product emitted. Six months later it was open on nobody's screen. The tiles answered questions nobody was asking, and the two numbers that actually preceded a decision — trial-to-paid rate and time to first value — were not among them. Analysis that does not end in a decision is reporting.

Quick check

A PM opens the analytics dashboard with no particular question in mind, just "to see what's there." What's the risk of this approach?

2

Quantitative vs. Qualitative Analysis

Quantitative analysis works with numbers, at scale. It is excellent at telling you what is happening, how often, and to how many people — and it is useless at telling you the reason. Qualitative analysis works with words and observation, in small numbers. It is excellent at why, and it cannot tell you how common anything is.

Qual alone produces vivid stories that may not represent anybody. Five interviews can be entirely truthful and entirely unrepresentative, and a memorable quote has a way of outliving the caveats attached to it.

The working pattern is to let them take turns. Quant tells you where to point the qual; qual tells you what the quant was actually measuring. Analytics located a 40% checkout drop within a day; five interviews explained it within a week — the field silently rejected card numbers containing spaces. Either half alone would have left the team stuck.

What & how much

Quantitative

Numbers at scale. Strong at measuring size and trend, weak at explaining why.

Why

Qualitative

Interviews, recordings, open-ended feedback. Strong at reasoning, weak at telling you how many.

Quant tells you where to look. Qual tells you why it's happening there.

Quantitative and qualitative, together A two-by-two grid crossing what a method tells you against how many people it covers. Analytics tells you what is happening at scale; interviews tell you why, in depth. Those two are highlighted, because the lesson is that they are used in turn rather than chosen between. Analytics what, at scale Survey with open text what, plus a hint of why Session replay what, for one person Interviews why, in depth what → why few people → many people
The two highlighted cells are diagonal on purpose: they share no axis, which is exactly why they are used together. The unhighlighted top-right is the tempting shortcut — one instrument that gestures at both — and it answers neither question well enough to act on.

Everyday example

Your car makes a noise. The dashboard tells you the temperature is high — that is the quantitative half, and it tells you what. The mechanic putting their ear to the engine is the qualitative half, and it tells you why. Neither one alone gets the car fixed.

The number that meant the opposite

Support contacts per user rose sharply after a release, and the team read it as a quality problem. Watching eight session recordings showed something else: a new feature was discoverable enough that people were asking how to use it well. The count was real; the interpretation needed the recordings.

Quick check

A funnel shows a 40% drop-off at the payment step, but the team doesn't know why users are leaving. What should they add to the analysis?

3

A Few Frameworks Worth Knowing

A framework is not a way of being clever. It is an order of operations — a checklist that stops you running the expensive test first, and stops you missing the obvious explanation because you were already attached to an interesting one.

Three are worth knowing well.

Segmentation: split the number before you try to explain it. A blended figure is an average of populations that may be moving in opposite directions, and averages conceal exactly the movement you are looking for.

Trend versus level: decide whether you care that the number is low or that it is falling. Those have different causes and different fixes, and teams frequently investigate one while worrying about the other.

Funnel decomposition: break an outcome into the steps that produce it, so “conversion is down” becomes “step three is down” — a question somebody can actually answer.

What makes any of them work is doing the cheap, broad checks before the deep ones. Segment before theorising. Check whether the tracking changed before concluding that behaviour did.

Most wrong analyses are not wrong because someone lacked a sophisticated method. They are wrong because an early step was skipped, and the sophisticated method was then applied very carefully to the wrong subset of the data.

Funnel Analysis

Break a multi-step process into stages and find where the biggest drop-off happens.

Cohort Analysis

Group users by a shared starting point (like signup week) and track how their behavior changes over time.

Segmentation

Slice a metric by a meaningful dimension to see if an aggregate number is hiding very different sub-stories.

Cheap checks before deep ones Three framework moves in order of cost: segment the number, separate trend from level, then decompose the funnel. Each is cheaper than the next and rules out whole categories of explanation. 1 Segment split before you average 2 Trend vs level falling, or just low? 3 Decompose which step, not the whole
Three checks, and the ordering is the point: each one costs minutes and rules out a whole category of explanation. Skip the first and everything after it is careful work on the wrong subset — which looks identical, from the inside, to careful work.

Everyday example

A doctor does not have one test, they have an order of tests: temperature first because it is cheap and narrows the field, scans later because they are expensive and specific. Analysis frameworks are that order — they stop you running the expensive test first.

Segment before you conclude

A blended conversion rate sat flat for two quarters and the team concluded nothing was working. Split by acquisition channel, one channel had doubled while another had halved, and the two movements had cancelled to a straight line. The framework did the work: the same data, cut once, said the opposite thing.

Quick check

A team's overall retention curve looks flat month over month, but they haven't checked whether newer signup cohorts are retaining better or worse than older ones. What analysis technique would reveal this?

4

Avoiding Bias in Analysis

Bias in analysis is rarely dishonesty. It is the ordinary human tendency to stop looking once you have found something that fits, and it survives good intentions comfortably.

Four are worth naming. Confirmation bias: you find what you set out to find, because you stopped searching when it appeared. Survivorship bias: you study only the people still present. Framing: the question carries its own answer. Cherry-picking the window: choosing the start date that makes the line behave.

Survivorship hides best, because the missing evidence leaves no trace in the data. Everyone still in your funnel is, by definition, someone the funnel did not remove — so studying them teaches you about survivable problems and nothing whatever about fatal ones. The people who hit the fatal problem are not there to be counted.

The defences are unglamorous and effective. Write down what you expect to find before looking, so you notice when you have found only that. Go looking specifically for the population absent from your data. And ask a colleague to argue the opposite case from the same numbers.

If they can do it easily, your conclusion was resting on the framing rather than on the evidence — which is much better to discover in a review than in front of the people who acted on it.

Trap 1

Correlation vs. Causation

Two things moving together doesn't mean one causes the other, a hidden third factor might drive both.

Trap 2

Cherry-Picking

Searching until you find the metric or time window that supports what you already believed.

Trap 3

Simpson's Paradox

A trend in several groups can reverse when the groups are combined, if their sizes differ enough.

Why the data you have is the data that survived Three layers showing who is visible in a dataset: everyone who might have used the product, those who got past the first barrier, and those still present to be measured. Only the last layer appears in analytics. Everyone who might have used it never appears in your data at all Reached the product some hit a fatal barrier here Still present to be measured this is your entire dataset what you can see
The bracket marks everything analytics can reach. Both layers above it are real people and neither leaves a trace — which is why survivorship hides better than the other biases: there is no anomaly to notice, no gap in the chart, just a clean dataset made entirely of people the funnel did not remove.

Everyday example

Ask "you liked that, didn't you?" and most people say yes. Ask "what did you think?" and you learn something. The question carries the answer inside it, and that is true of a query as much as an interview.

Survivorship, in aircraft and in funnels

In the Second World War, analysts studying returning bombers proposed armouring the areas with the most bullet holes. Abraham Wald pointed out the flaw: those were the planes that made it home, so the holes showed where a bomber could survive damage. The armour belonged where the returning planes had no holes at all. Product analytics has the same shape — the users still in your funnel are the ones the funnel did not remove.

Quick check

A report claims "ice cream sales cause more drownings" because both rise together every summer. What's the analytical error?

Drill what you learned
Scenario 1 easy

A PM is asked "how is the product doing?" with no further detail, and immediately starts pulling every metric they can find into one giant report.

What's a better first step?

Scenario 2 easy

A funnel analysis shows the biggest drop-off is between "add to cart" and "enter shipping info," but the team doesn't know if it's a UX problem or a shipping-cost surprise.

What should come next?

Scenario 3 easy

A cohort analysis shows the March 2025 signup cohort retaining much better at 90 days than the January 2025 cohort.

What's a reasonable next step?

Scenario 4 easy

A report says "users on our premium plan are happier, because premium-plan NPS is 20 points higher than free-plan NPS."

What alternative explanation should the analyst consider?

Scenario 5 easy

A dashboard shows "conversion rate" as one aggregate number for both desktop and mobile combined, and it's been flat for months.

What analysis technique might reveal something the aggregate is hiding?

Scenario 6 medium

An analyst notices Hospital A has a higher survival rate than Hospital B for both mild AND severe cases individually, yet Hospital B has a higher overall survival rate.

What's the likely explanation?

Scenario 7 medium

An analyst keeps re-slicing a flat metric by different time windows until they find a 3-day period where it happened to spike, then reports "engagement is up."

What's the analytical problem?

Scenario 8 medium

A PM wants to know why enterprise customers churn at a higher rate than small-business customers.

What analysis approach best answers "why," beyond just confirming the rate difference?

Scenario 9 medium

A team segments a metric by acquisition channel and finds one channel's users convert twice as well as others, then immediately doubles that channel's budget.

What should they check before doing this?

Scenario 10 medium

An analyst reports "feature X caused a 10% revenue increase" based solely on revenue rising in the month after feature X launched.

What's missing from this conclusion?

Scenario 11 hard

A retention chart shows an overall improving trend, but a closer cohort breakdown shows every individual cohort is actually declining.

What's happening?

Scenario 12 hard

A team wants to understand both how many users are affected by a bug and why it's happening to them.

What combination of methods fits best?

Scenario 13 hard

A metric review meeting spends 90% of its time debating a metric that moved by 0.1%, within its normal week-to-week noise range.

What's the issue?

Scenario 14 hard

An analyst wants to compare this quarter's conversion rate to last quarter's, but pricing changed in between the two periods.

What should the analysis account for?

Scenario 15 hard

A team concludes "our new onboarding is a failure" based on one week of data, right after a major unrelated outage occurred during that same week.

What's the analytical error?

Drilled it. Now apply it to a real situation.

Put it to work
From lesson 1

Target knew before the family did

Target

Target's analytics team built a pregnancy-prediction score from purchase history — unscented lotion, certain supplements, cotton wool — and used it to time baby-product mailers. It worked. It worked well enough that a father complained about coupons sent to his teenage daughter, and later found out she was pregnant.

The analysis was correct. The decision it drove was the problem.

What does this say about turning data into a decision?

From lesson 2

Two studies, one number, opposite conclusions

A B2B expense tool

Support tickets about the receipt scanner are up 40% this quarter. Two people investigate.

The analyst pulls the numbers: scan failures are flat, ticket volume is up, and the increase is concentrated in accounts that joined in the last 90 days. The researcher runs six interviews with people who raised tickets.

Work the two sources against each other.

  1. Step 1 of 3

    The quantitative read says scan failures are flat. What has it established?

    You know where it is happening and that it is not a reliability regression. You do not know what is going on.

  2. Step 2 of 3

    The interviews find that all six expected the scanner to read multi-page receipts, and it only reads the first page. What has that established?

    So: go back to the data and count how many tickets involve receipts of more than one page.

  3. Step 3 of 3

    That count comes back at 71% of the increase. What have the two methods done between them?

    71% of a 40% rise is a number you can take to a prioritisation meeting. Six interviews are not.

From lesson 3

Five whys, on a monument

The Jefferson Memorial

The Jefferson Memorial's stonework was deteriorating faster than the other monuments on the National Mall. The obvious answer was more frequent, gentler cleaning.

The chain that was actually followed is the most-cited example of the technique: the stone was damaged by harsh detergent, used because of heavy bird droppings, which were there because of the spiders the birds fed on, which were there because of the midges the spiders fed on, which swarmed at dusk because the floodlights came on earlier than at the other monuments.

Put the chain back in order, from the symptom to the root.

Drag the rows, or use the arrows, then check.

  1. The stone is eroding The symptom — the thing anyone can see, and the thing the obvious fix would have addressed.
  2. It is being cleaned with harsh detergent, frequently The proximate cause. Stop here and you get gentler detergent, more often — treating the erosion without touching what produces it.
  3. There are heavy bird droppings to clean off One level down, and the first answer that explains the cleaning rather than describing it.
  4. The birds are there because of the spiders Now the chain has left the building entirely, which is usually the sign you are getting somewhere.
  5. The spiders are there because of the midges, which swarm when the floodlights come on The root: a lighting schedule. It is the only link in the chain where a cheap change breaks the whole sequence.
From lesson 4

The survey that only asked the survivors

A project-management SaaS

Churn is rising. The team sends an in-app survey asking what would make the product more useful, and gets 900 responses. The top request, by a distance, is more granular permissions.

Two quarters of work later, permissions ship. Churn does not move.

Which biases are at work here?

Select all that apply — there are 3 to find.

Notification