The growth experiment that wastes the most time

The most common growth experiment failure mode is not running a poorly designed test. It is running a correctly designed, statistically valid test on an outcome metric so disconnected from the business that a 30% improvement in it would change nothing. A/B testing CTA button colours and hero image variants are not growth experiments. They are busywork with confidence intervals.

The leverage problem no one talks about

Growth teams get evaluated on experiment velocity. How many tests did you ship this quarter? What was your win rate? The implication is that more experiments, run faster, produce better outcomes. This is approximately backwards. The number of experiments run is a near-useless metric. The leverage of the experiments run is the only thing that matters, and it is almost never tracked.

Leverage, for a growth experiment, has a specific meaning: what is the maximum possible business impact if the experiment produces the strongest plausible result? An experiment testing whether a green CTA button outperforms an orange one on a page that converts at 4% might produce a 10–15% relative improvement in clicks — taking the conversion rate from 4.0% to 4.5%. If that page drives 2,000 qualified visitors per month and the average contract value is $500, the annual revenue impact of a successful test is approximately $60,000. This is not nothing. But it consumed the same engineering, design, and analytical bandwidth as a test that could have doubled the conversion rate on a higher-traffic, higher-ACV page.

The teams running experiments on button colours and email subject lines are not doing this because they think these tests are high-value. They are doing it because these tests are easy to scope, easy to build, and easy to call conclusive. There is a powerful organisational incentive to run low-leverage experiments fast rather than high-leverage experiments slowly. Experiment velocity looks good in a sprint review. The revenue impact of a high-leverage experiment takes a quarter to materialise and may require cross-functional coordination that is hard to justify in a two-week sprint cycle.

The highest-leverage growth experiments and why they are harder to run

The experiments that produce step-change improvements in conversion rate are almost always positioned in one of three places: the value proposition itself, the first-use experience, or the upgrade trigger. These are harder to test than button colours because they require substantive changes to product and messaging — changes that require sign-off from more stakeholders and produce noisier results. But the effect sizes are an order of magnitude larger.

Value proposition experiments. Testing different framings of what the product does and who it is for. Not "we changed the headline from 'Grow faster' to 'More pipeline, less effort'" — those are copy tests. Value proposition experiments change which benefit is foregrounded, which customer segment is addressed, and which competitive alternative is implicitly displaced. A well-designed value proposition test on a high-traffic landing page can produce large improvements in qualified click-through. A button colour test cannot.

First-use experience experiments. The first three minutes a user spends in your product determines more about their retention trajectory than anything that happens in the following three months. Most products have never been systematically tested at the first-use stage. They have been designed by the people who understand the product best, for the people who understand the product best. Testing different first-use flows — different empty states, different guided tours, different first-action prompts — can produce double-digit improvements in week-one activation rates, which compounds directly into 90-day retention.

Upgrade trigger experiments. For products with a paywall, the placement of the upgrade prompt, the framing of the value gap, and the timing of the offer are all testable — and all vastly more impactful than the colour of the upgrade button. Testing whether to surface the paywall after a specific usage event versus after a time-based trigger versus after a collaboration event can produce step-change conversion improvements. Testing the button colour on the paywall modal produces, at best, a few percent.

How to diagnose your experiment portfolio

The most useful diagnostic for a growth team's experiment programme is a leverage audit: for every experiment run in the last two quarters, estimate the annual revenue impact of the result — win, loss, or inconclusive. What was the range of outcomes? What was the maximum upside if the test had won at a 2x effect size? What would it have taken to double the average leverage of the experiments run?

Many teams that do this exercise find that most of their experiment time went to tests where the maximum plausible upside was small — say, under $200K in annual revenue impact — and only a sliver went to tests where the upside was $1M+. The right ratio, for a growth team working at a company with $3–10M ARR, is roughly the inverse: the majority of experiment time should be on tests with $500K+ maximum upside, and the low-leverage tests should be delegated or eliminated.

This reallocation is uncomfortable. High-leverage experiments take longer to design, longer to run to significance, and longer to ship. They require the growth team to work more closely with product, design, and sometimes engineering leadership. They produce results that are harder to present in a sprint review because the feedback cycle is four to six weeks rather than two. None of this is a reason not to run them. It is a reason to change how growth teams are evaluated and how experiments are prioritised.

The prioritisation framework that actually works

Before any experiment enters the roadmap, answer four questions: What metric does this test move? What is the maximum plausible effect size? What is the annual revenue impact of that maximum effect? Who needs to approve and ship the change if it wins? If the maximum revenue impact is below your team's threshold (set this explicitly — $300K, $500K, whatever is right for your ARR), it goes on the backlog. If the approval and shipping path is unclear, resolve it before the test enters the roadmap. The experiment backlog is not a place where ambiguous tests wait to be approved. It is a graveyard for tests that were never properly scoped.

• 01
  Require a maximum-impact estimate before any experiment enters the roadmap. What is the annual revenue impact if the test wins at a 2x effect size? Set a minimum threshold. Any test that cannot clear it goes to the backlog or gets dropped entirely.
• 02
  Audit your last two quarters of experiments for leverage. What percentage of experiment time was spent on tests where the maximum upside was less than $200K? What would it have taken to double the average leverage? Run this audit quarterly and present the results to the leadership team.
• 03
  Prioritise first-use experience experiments above all else. Week-one activation rate is the highest-leverage conversion metric in almost any SaaS product. Improvements compound into 90-day retention, which compounds into NRR. A 20% improvement in week-one activation is worth more than a 20% improvement in CTA click-through rate.
• 04
  Separate value proposition tests from copy tests. Copy tests change words. Value proposition tests change which benefit is foregrounded and which customer is being addressed. The second type typically produces much larger effect sizes and is run far less frequently because it requires more cross-functional alignment.
• 05
  Never run an experiment you cannot reach significance on within six weeks. If your traffic volume requires eight to twelve weeks to reach significance at 80% power and a 5% minimum detectable effect, you do not have enough data to run this experiment. Use qualitative research instead: customer interviews, session recordings, and five-second tests tell you more in two weeks than an underpowered A/B test does in three months.

Frequently asked questions.