The metrics that tell you your startup is lying to itself

Vanity metrics have been written about so thoroughly that founders now know to avoid them. What hasn't been written about enough are the legitimate-looking metrics — the ones that appear in serious board decks, that sophisticated investors ask about, that your analytics tool surfaces by default — that are quietly hiding the story you actually need to see.

The problem with being almost right.

The danger of a vanity metric is obvious in retrospect. Page views, app downloads, registered users — everyone now knows these numbers can rise while the underlying business deteriorates. The literature on this is extensive. The antibody has been developed.

The danger of a sophisticated misleading metric is much harder to see. These metrics are not wrong. The data is accurate. The calculation is correct. The problem is what they are hiding — the assumptions baked into the aggregation, the channels masked by blending, the cohorts smoothed away by averaging. They feel like signal because they are constructed from real data. They function as noise because they average away exactly the information you need.

Leadership teams are often genuinely convinced the business is on track, because the metrics they track say so — and then get blindsided quarters later, when the reality that had been averaging itself out finally becomes impossible to ignore. The problem is rarely dishonesty. The problem is metric design.

Blended CAC: the channel problem hiding in plain sight.

Blended customer acquisition cost is calculated simply: divide total acquisition spend by total new customers acquired in a period. It is on almost every growth dashboard. It is almost always misleading.

Here is what blended CAC hides. Suppose you have three acquisition channels: organic search (high volume, near-zero marginal cost), paid social (medium volume, moderate CAC), and outbound sales (low volume, very high CAC). Your blended CAC looks reasonable — let's say $45. Your organic channel is doing most of the heavy lifting and subsidising the other two. But if someone asks whether your paid social CAC is sustainable, or whether the outbound motion has unit economics that work at scale, blended CAC cannot answer that question. It has averaged the answer away.

Now here is what makes this dangerous at growth stage. Organic channels saturate. When you have captured most of the high-intent search audience for your category, the marginal organic customer gets more expensive to acquire. If your paid and outbound channels have CACs that don't work at scale, blended CAC will rise sharply — and leadership will be surprised, because the blended number gave no warning.

What to track instead: channel-segmented CAC with a clear view of which channels are subsidising which, and a headroom analysis on your lowest-cost channels — how much more volume is there before the economics change? This is not harder to calculate. It is just less flattering to present.

Engagement metrics that mask retention decay.

DAU, WAU, session length, feature usage — these are all legitimate measures of user engagement. They are also the metrics most commonly used to paper over declining retention, because aggregate engagement can rise even as cohort-level retention deteriorates.

The mechanism is simple. If you are acquiring new cohorts faster than old cohorts are churning, your aggregate DAU rises. New cohorts typically have high initial engagement. So aggregate engagement rises, while the retention curve of each individual cohort is declining. The aggregate number looks healthy. The underlying business is a leaky bucket that is filling faster than it is draining — for now.

The test is straightforward: plot retention curves by acquisition cohort. What is the three-month retention rate for your January cohort versus your October cohort versus your cohort from two years ago? If the curves are deteriorating — if month-three retention for recent cohorts is lower than it was for older ones — you have a product or market fit problem that aggregate engagement metrics are hiding.

The pattern is common in fintech and B2B SaaS alike: rising DAU, declining cohort retention, and a retention problem discovered far later than it should have been, because the aggregate metric was structurally incapable of surfacing it.

NPS: measuring the wrong moment at the wrong altitude.

Net Promoter Score is one of the most widely used and most consistently misapplied metrics in tech. The core problem is this: NPS measures intent in a moment — specifically, how a customer feels about you at the moment they receive the survey — and presents that as a proxy for loyalty and retention. These are not the same thing.

A customer can give you a nine and churn three weeks later. A customer can give you a six and be your highest-value account for the next three years. Intent expressed in a survey moment does not predict behaviour over time, and the link between NPS and actual retention is often weak enough that treating NPS as a retention indicator is a mistake, not just a framing preference.

There is a second problem: NPS surveys are typically triggered by a recent positive experience — a successful support interaction, a new feature launch, a renewal conversation. The score you are measuring is often the quality of that recent touchpoint, not the baseline health of the customer relationship. A customer who had a great support experience last week and is about to churn still gives you an eight.

What to track instead: actual retention rate by cohort, product usage of core value features (not total sessions, but the specific actions that correlate with contract renewal), and qualitative signals from churned customers — not from satisfied ones. Your churned customers know things about your product that your NPS survey will never surface.

Revenue metrics that hide the mix problem.

Total ARR growth is the headline number every board deck leads with, and it is often the most misleading aggregate of all. The problem is mix. ARR growth can be composed very differently depending on whether it is driven by new logo acquisition, expansion from existing customers, or price increases — and these compositions have radically different implications for the durability of growth.

ARR growth driven primarily by expansion revenue (existing customers buying more) is structurally different from ARR growth driven by new logos. Expansion revenue is high-margin and has no acquisition cost. New logo acquisition has CAC, ramp time, and churn risk. A company with flat new logo acquisition but strong expansion looks healthy on an ARR growth chart — until the expansion pool saturates because you are not adding enough new customers to expand into.

Similarly, gross revenue growth hides net revenue retention. If your gross ARR is growing fifteen percent but your net revenue retention (which accounts for churn) is ninety percent, you are working very hard to fill a bucket that is losing ten percent of its volume every year. Compare that to a company growing gross ARR at ten percent with a hundred and fifteen percent net revenue retention — the second company is building a more durable business by a significant margin.

The five metrics habits that separate honest teams from self-deceived ones.

• 01
  Never present a blended number without the components. Blended CAC, blended churn, blended ARPU — these are summary lines, not insights. Every blended number in your board deck should be accompanied by the channel or segment breakdown that produced it. If the breakdown isn't there, the number isn't telling you anything actionable.
• 02
  Plot every retention metric by cohort, not in aggregate. Aggregate retention metrics are structurally incapable of surfacing cohort-level deterioration. The ten minutes it takes to cut your retention data by cohort is the highest-ROI analytical investment in your calendar. If you only have one cohort chart in your company, make it month-three retention cohort-over-cohort.
• 03
  Ask what the metric cannot see. Before you present any metric in a leadership review, ask: what scenario would produce this number while the underlying business deteriorated? If you can construct a plausible answer, you are missing a metric. This is not paranoia — it is analytical hygiene.
• 04
  Track leading indicators, not just lagging ones. ARR, revenue, and NPS are all lagging indicators — they tell you what already happened. The metrics that tell you what will happen three months from now are leading indicators: product activation rate for new users, number of power users per account, support ticket volume by category. Identify your leading indicators and track them with the same rigour as your headline numbers.
• 05
  Include a slide for what the data doesn't tell you. The most useful thing you can put in a board deck or leadership review is an honest account of the questions your current metrics cannot answer. This forces the question of what data you need to collect, and it builds credibility with investors who know that every metrics deck has a blind spot.

Frequently asked questions.