How to answer the metric dropped interview question product managers get, worked on a 15 percent drop

What is the metric dropped question really testing?

The metric dropped question tests whether you can get from one alarming number to a cause you could act on, in a fixed order, while talking to someone who holds the data. A worked example in the Inflection Point newsletter's guide to root cause analysis problems shows the shape: a candidate is told LinkedIn has "seen a 20% drop in profile views of LinkedIn's high-profile users." After a run of clarifying questions, the drop turns out to be concentrated among government and public sector profiles, and the cause is not a product problem at all: views had spiked around an election six months earlier and were returning to their old level.

That example shows the two things interviewers listen for. The first is order: the candidate asked how the metric was defined and whether the calculation had changed before proposing any cause. The second is collaboration. The same guide puts it directly: "the RCA interview is always a collaborative discussion; always keep looking for hints from the interviewer." The interviewer is a data source. Every question you ask is a query you are running, and a good question removes half the possible causes.

Time is short. Ben Erez, a former Facebook PM, writes in Lenny's Newsletter that an analytical thinking interview "normally lasts 45 minutes", leaving "only about 35 minutes of actual working time." A diagnosis that wanders through every possible cause runs out of time before it reaches one.

What are the six steps of a metric drop diagnosis?

A metric drop diagnosis runs in six steps, and the order matters more than the detail inside each one.

1. Clarify the definition and the window. What exactly is counted, over what period, compared with what, and was the drop sudden or gradual?
2. Rule out instrumentation. Did tracking, the event pipeline or the metric's definition change in the window?
3. Rule out seasonality and the calendar. Does the same period last year show the same dip? Was there a holiday, a school break or a day-of-week effect?
4. Segment the drop. Cut it by platform, acquisition cohort, geography and funnel step, and find where it concentrates.
5. Form two or three hypotheses. Each one should be specific and come with the data that would separate it from the others.
6. Say what you would check first, and why, then the action for each outcome. Cheapest decisive check first.

Steps 2 and 3 exist because the most likely explanation for a sudden change is often not a product change at all. Steps 4 and 5 are where most of the interview's time goes, and where most of the score is earned.

Which clarifying questions should you ask first?

Ask the clarifying questions whose answers change your next step, and skip the ones that only sound thorough. Erez's test for a good metric applies here too: "If you can't describe your metric to a data scientist in a way that they could run a query with, it's not a useful metric." Your clarifying questions should leave you able to write that query.

Question	Why it matters	What a surprising answer changes
How exactly is the metric defined?	"Active" or "retained" can mean several different events	You may be diagnosing a definition, not a behaviour
Is 15 percent relative or in points?	40 to 34 percent is a 15 percent relative drop and a 6 point drop	The size of the problem, and which segments can explain it
Over what window, compared with what?	Week on week, against last year, against a forecast	A gradual decline points to different causes than a step change
Sudden or gradual?	Step changes usually have a dated cause: a release, a campaign, an outage	You can line the drop up against a change log
Did anything else move?	Related metrics moving together narrow the cause	Sign-ups up while retention falls suggests a mix change

Three or four of these are enough. Then say your plan in one sentence, so the interviewer can follow and redirect you: "I'll rule out tracking and seasonality first, then segment the drop, then give you two or three hypotheses and what I'd check first."

How do you rule out tracking errors and seasonality before looking for causes?

Rule out tracking errors and seasonality first, because they are the cheapest explanations to test and the most embarrassing to miss. Twyman's law, named after the media and market researcher Tony Twyman, holds that "any figure that looks interesting or different is usually wrong." The Wikipedia entry gives the product version: if a software company's user count doubles overnight, "the most likely explanation is a bug in logging". A sudden 15 percent drop deserves the same suspicion.

For instrumentation, ask three things. Did a release in the window change how the event is sent (a new analytics library, a renamed event, a consent banner that blocks tracking)? Does a second source agree, such as server-side records or the database, rather than client-side analytics? Did anyone change the dashboard's definition or filters?

For seasonality, compare the same weeks last year, check for holidays and term dates in your main markets, and check whether the drop is concentrated on particular days of the week. If last year shows the same dip, say so and ask whether the interviewer wants you to go on; often the case is about recognising exactly that.

How should you segment the drop?

Segment by four cuts, one at a time, and look for where the drop concentrates rather than where it merely appears.

• Platform: iOS, Android, web. A drop on one platform points at a release or a store change.
• Cohort: when and how users arrived, by sign-up week and acquisition source. A drop in new users only points at onboarding or acquisition quality.
• Geography: country or region. A drop in one market points at a local event, competitor or outage.
• Funnel step: each step on the way to the metric. The step whose conversion falls is where the cause lives.

Leland's guide to the product execution interview gives the phrasing to aim for: instead of repeating the headline number, say "The drop was sharpest among newly registered users, while returning users were stable." That sentence has located the problem.

One trap needs naming: mix shift. If a lower-performing segment grows as a share of users, the total falls even when every segment's own rate holds steady. Whenever acquisition changed in the window, check how much of the drop the new mix explains on its own.

What does a complete answer to "week-1 retention dropped 15 percent" sound like?

The worked example below is original, with the interviewer's answers included so the whole dialogue is visible.

The prompt. "You're the PM for a habit-tracking app. Week-1 retention for new users dropped 15 percent over the last two weeks. What's going on?"

Clarify. The candidate asks four questions:

• "How is week-1 retention defined?" The share of a week's new sign-ups who complete at least one habit on days 2 to 7.
• "Is 15 percent relative?" Yes. It was about 40 percent across the previous four weekly cohorts; the last two are at 34.
• "Sudden or gradual?" Sudden. The first low cohort is the one that signed up from Monday two weeks ago.
• "Did anything else move?" Sign-ups are up about 10 percent. Growth started a paid social campaign the same week.

Plan. "A sudden drop with a start date, and sign-ups up at the same time. I'll check tracking and the calendar, then segment by platform, source, geography and funnel step."

Rule out. "Do server-side habit completions show the same drop as the analytics dashboard?" Yes. "Same weeks last year?" Flat, within a point. "So the drop is real and not seasonal."

Segment. The candidate asks for each cut and sketches the table:

Cut	Segment	Share of sign-ups (before, now)	Week-1 retention (before, now)
Platform	iOS	55%, 52%	42%, 41%
Platform	Android	45%, 48%	38%, 26%
Source	Organic	70%, 62%	44%, 38%
Source	Paid social	30%, 38%	31%, 28%
Geography	Home market	60%, 58%	41%, 35%
Geography	Rest of Europe	40%, 42%	39%, 33%
Funnel (Android)	Created a first habit	n/a	78%, 77%
Funnel (Android)	Allowed reminders	n/a	64%, 41%
Funnel (iOS)	Allowed reminders	n/a	61%, 60%

"Each cut adds up to the total: the platform rows give about 40 percent before and 34 now, and so do the source and geography rows. Geography is flat, so it isn't a market problem. The drop sits almost entirely on Android, and inside Android it sits at one step: the share of new users allowing reminders fell from 64 to 41 percent while first-habit creation didn't move."

"The source cut matters too. Paid social grew from 30 to 38 percent of sign-ups and retains worse. If I hold each source at its old rate and apply the new mix, retention would be about 39 percent, so the mix explains roughly one point of the six. Both sources fell, so the mix isn't the main cause."

Hypotheses. The candidate gives three, each with the data that separates it:

1. An Android release changed the reminder prompt. "Did an Android version ship when the drop started?" Yes, 5.12, that Monday. It moved the reminder permission prompt to before the first habit is created. Separating data: retention and reminder opt-in for Android users on 5.12 against those still on 5.11 in the same weeks.
2. The paid campaign brought lower-intent users, mostly on Android. Separating data: retention for organic Android users alone. If organic Android fell as much, the campaign is not the main cause.
3. Reminders stopped arriving on Android. Separating data: reminders sent against reminders delivered and opened, among users who allowed them.

Check first, and why. "I'd check the first one first. The timing matches a dated release, the funnel puts the drop exactly at the step that release changed, and comparing users on 5.12 with users on 5.11 is one query."

Action per outcome. "If 5.12 users opt in and retain at the lower rates and 5.11 users don't, I'd restore the old prompt order behind a flag now, then test prompt timing properly as an experiment. If there's no version difference, I'd look at reminder delivery next. If neither explains it, I'd take the mix finding to growth, and ask for retention to be reported by source so the next campaign can't hide a drop or fake a recovery."

The answer never named a cause before the data pointed at one, and it finished with three possible next moves rather than one confident guess.

Which mistakes make a diagnosis answer fail?

Most failed diagnosis answers break the order rather than miss a cause:

1. Causes first. Listing "a competitor launched, a bug, seasonality, a bad release" in the first minute. The list may contain the answer, but nothing shows how you would find it.
2. No definition. Diagnosing "retention" without asking what it counts, then discovering halfway through that it measured something else.
3. Segmenting everything. Asking for every cut at once instead of the one most likely to concentrate the drop, then running out of time.
4. Hypotheses with no test. "It could be the release" with no data that would confirm or rule it out.
5. No close. Ending on "I'd need to look at the data" instead of what you would check first and what you would do with each answer.

Is the root cause question still worth preparing for?

Prepare for the root cause question, but not only for it. Erez writes that analytical thinking interviews "can sometimes include other question types, such as debugging/root cause analysis questions and estimation questions, but these are becoming less common." His own guide spends its main steps on product rationale, a metric framework and goal-setting instead.

The counterweight is that the diagnosis is a daily part of the job. A PM who cannot tell a tracking error from a real behaviour change will spend weeks fixing a problem that does not exist. The six steps also carry over to the metrics questions that are still common: choosing a metric well, as in how to measure feature success, is the same skill as diagnosing one, run in the other direction. If the follow-up asks whether an experiment result is real, statistical vs practical significance covers that call, and retention analysis for product managers covers the cohort tables behind the worked example. For the trade-off half of the same round, see how to answer execution interview questions.

Practise analytics and diagnosis cases with feedback

Builders Camp's Product Interviewing bootcamp runs across 2 weeks. Its public curriculum covers analytics and experimentation cases (metrics, diagnosis, A/B testing and decisions with imperfect data) alongside product sense, execution, strategy, and a mock interview and improvement loop.

See the Product Interviewing bootcamp