--- title: "Retention Cohort Analysis Practice Exercise" description: "It's 2011. Instagram has 1 million users and a retention crisis. Read the cohort data, pick the highest-impact fix, and design one experiment. 60 minutes." canonical_url: "https://builderscamp.com/guides/challenges/growth-for-product-managers-retention-cohort-diagnosis" date_published: "2026-09-16" date_modified: "2026-09-16" author: "Andre Albuquerque" publisher: "Builders Camp" guide_class: "challenges" --- # Retention cohort analysis practice exercise **TL;DR:** This exercise puts you at a fast-growing app in its earliest days, past 1 million users but losing most new signups within a week. You read 90 days of cohort data, choose between two competing problems with only one experiment slot available, design a rigorous A/B test, and defend the call to a CEO who wants an answer before standup. ## The scenario It is early 2011. Instagram has just crossed 1 million users and growth looks strong from the outside. Underneath the headline number, most people who sign up do not come back after their first week. The person running the company pulls you aside directly: the company is bleeding users, and someone needs to say what is going on and what to fix first. You are handed 90 days of cohort data and a single, hard constraint: engineering has capacity for exactly one experiment this sprint. Choose the wrong problem and two weeks are gone with nothing to show for it. Choose the right one and you might crack the retention problem that everything else depends on. The data shows two real patterns worth attacking. The overall drop from week one to week two is steep, and it affects every single new user regardless of what they did when they signed up. Separately, only a minority of new users follow enough accounts in their first session to hit a threshold that predicts much stronger long-term retention, and the gap in outcomes between that group and everyone else, by week 12, is wide. Both are real. You can only chase one this sprint. ## What you are asked to do The exercise moves through five connected steps: - Read the cohort data and pull out the three numbers that would actually anchor a presentation to the CEO: the single biggest drop in the funnel, the retention gap between the best and worst segments at week 12, and the number that represents the single biggest opportunity. - Compare the two competing problems, the steep early drop-off and the low first-session follow rate, on scale, on how confident you are that the relationship is causal, and on how easy each would actually be to intervene on, then choose one. - Write three hypotheses for why users are churning at the point you chose, each naming the specific moment in the product, the expectation that is not being met, and what that implies about what to change. - Design a rigorous A/B test for your strongest hypothesis, specifying control, variant, primary metric, minimum detectable effect, run length, and how you would call a winner. - Deliver the decision to the CEO in three sentences or fewer, then name the biggest risk of being wrong and what the data would look like if that risk played out. ## What a strong answer covers The exercise's own objectives are the standard a strong submission has to meet: - Are the three headline numbers pulled correctly and stated precisely, with what each one actually means for the business, rather than a vague summary of "retention looks bad"? - Does the comparison between the two competing problems weigh scale, causal confidence, and ease of intervention explicitly, rather than picking the more dramatic-sounding number by default? - Do the three hypotheses each name a specific moment in the product and a specific unmet expectation, rather than a generic statement that onboarding needs improvement? - Does the experiment design include a real minimum detectable effect and a defined way to call a winner, specific enough that an engineering team could actually build and ship it? - Does the risk statement describe what the data would actually look like if the chosen hypothesis turns out to be wrong, rather than a generic acknowledgment that experiments sometimes fail? ## Skills this exercise practises Reading cohort retention data for the number that actually matters, not just the one that is easiest to explain. Choosing between two real, competing problems under a hard resource constraint instead of trying to solve both. Turning a diagnosis into hypotheses specific enough to design an experiment around. Designing an A/B test rigorous enough to survive scrutiny before it gets a sprint. These map onto the bootcamp's own curriculum on retention and engagement loops, growth measurement and cohort analysis, and the experimentation system. For the mechanics of reading a partial or mixed A/B test result once your experiment is running, the [A/B test results exercise](https://builderscamp.com/guides/challenges/ab-testing-for-product-managers-ab-test-results) from A/B Testing for Product Managers is a natural next exercise. For the metric discipline behind choosing what to measure in the first place, see [north star metric examples](https://builderscamp.com/guides/other/north-star-metric-examples), and for prioritizing the fix once the experiment has an answer, the [RICE prioritization framework](https://builderscamp.com/guides/other/rice-prioritization-framework). ## Which bootcamp this comes from This exercise is the practical challenge from [Growth for Product Managers](https://builderscamp.com/bootcamps/growth-product-manager), a two week bootcamp on Builders Camp with four live sessions covering growth foundations, retention and engagement loops, acquisition loops, and growth measurement. Completing the practical challenge counts toward the bootcamp's completion requirement and its certificate, alongside the certification quiz. Builders Camp runs this bootcamp both live and self-paced, included with the Builders Camp Membership alongside every other bootcamp, track, and masterclass. ## Frequently asked questions ### What is the scenario behind this growth exercise? It is early 2011 and a fast-growing photo sharing app has just crossed 1 million users, but most new users do not come back after their first week. You have 90 days of cohort data, one experiment slot for the sprint, and have to decide what to fix first. ### Do I need real cohort analysis experience to attempt this? Basic comfort reading a retention curve and a percentage drop-off is enough. The exercise gives you the numbers directly and is built around the decision of what to do with them, not the mechanics of building a cohort table from raw logs. ### Why can you only choose one problem to attack? Because the exercise mirrors a real constraint: engineering has capacity for exactly one experiment that sprint. Choosing between the biggest absolute drop-off and a smaller but higher-impact segment gap is the actual skill being tested, not identifying that both problems exist. ### What is the hardest part of designing the experiment? Naming a real minimum detectable effect and a clear way to call a winner, rather than a vague plan to 'test onboarding improvements.' The exercise specifically asks for a rigorous design that would survive an engineering team asking why it deserves the sprint. ### How long does the exercise take? About 60 minutes, rated intermediate difficulty, inside a two week bootcamp with four live sessions. ### Does completing it count toward a certificate? Yes. Finishing the practical challenge counts toward completing the Growth for Product Managers bootcamp on Builders Camp, alongside the certification quiz, and the bootcamp issues a certificate on completion. ## Sources - [Builders Camp: Growth for Product Managers bootcamp page](https://builderscamp.com/bootcamps/growth-product-manager) ## How this guide was made Researched from Builders Camp's bootcamp, track and masterclass material and the sources listed on this page, drafted with AI, and fact-checked against every source cited.