Retention curves lie for the first two weeks
Practice
If you've ever watched a retention curve for a cohort that's only a week or two old and felt encouraged by a number that later dropped, you've run into one of the most common and most avoidable mistakes in early-stage product analytics.
The small-sample problem
A cohort of 40 signups from launch week will show wild swings in its day-7 retention as individual users churn or come back. Losing three users out of 40 moves the retention percentage by 7.5 points. The same three users lost from a cohort of 4,000 barely register. Early cohorts are noisy by definition, and noise looks like a trend if you're watching closely enough.
The excitement bias
Teams tend to check retention obsessively right after launch, which means the first cohorts get read at exactly the moment they're least statistically reliable, and read by people who most want the number to be good. That combination produces a lot of premature celebration and a lot of premature panic, often about the same underlying cohort a week apart.
What we recommend instead
Don't trust a retention number until the cohort has at least a few hundred users in it, or has had time for the curve to actually flatten out past day 30. Before that, treat the number as a rough signal, not a verdict. Compare shapes across cohorts (is the curve flattening at all, regardless of where) rather than fixating on the exact percentage of any single early cohort.
Nocturna shows a confidence indicator next to any cohort under 200 users specifically so this mistake is harder to make by accident. It's a small thing, but it's saved a few of our own users from shipping a retention-based decision off a sample too small to support it.
