The user interview mistake that invalidates your research

Most user research produces confirmation, not insight. The problem isn't that teams skip research — it's that the techniques they use are systematically designed to tell them what they already believe. Structured interview guides, post-session surveys, NPS after a demo: these are rituals of due diligence, not instruments of discovery. Here is what actually produces signal.

The research that confirms everything.

Picture a typical research readout. A PM presents the findings from twelve user interviews in a forty-two-slide deck. The conclusion is that users "strongly valued simplicity," "wanted faster onboarding," and "found the current workflow confusing." The recommendations are to simplify the interface, improve onboarding, and reduce workflow friction. The product team nods. The deck is saved to the shared drive. Six months later, the same interface, the same onboarding, and the same workflow ship with minor surface changes.

This is not an unusual story. It is the common story. Research is conducted. A report is written. The report confirms the intuitions everyone arrived with. The recommendations are agreed upon. Nothing changes, because the research didn't produce anything surprising enough to force a choice that hadn't already been made. Everyone feels rigorous. Nobody has learned anything.

The failure isn't that the PM is bad at research. It is that the research was designed — structurally, in the question guide and the interview format and the analysis methodology — to produce confirmation rather than surprise. And confirmation is comfortable. It is safe. It is easy to present to leadership. It is also, almost by definition, useless. If research only confirms what you already believe, it is not research. It is elaborate agreement.

The bias built into how you ask questions.

The most damaging mistake in user interviews is asking about the future instead of asking about the past. "Would you use a feature that did X?" "How important is Y to you?" "On a scale of one to ten, how much would this improve your workflow?" These questions feel like data. They are not. They are social transactions.

When you ask someone "would you use this?", they are not predicting their behaviour. They are calculating what you want to hear and adjusting for social pressure. If you've just shown them a product, they know you made it. They know you care whether it succeeds. They will tell you they would use it because the alternative — telling you to your face that they wouldn't — requires a degree of social confrontation that most people prefer to avoid. The academic literature on this is extensive. The practical implication is simple: everything a user tells you about their hypothetical future behaviour is unreliable.

What is reliable is what users have actually done. Past behaviour — specific, recalled, observable — is the only material that produces usable signal. "Walk me through the last time you had to do X" produces real information. "How do you usually handle Y?" begins to produce real information, though it's one step more abstract. "Would you prefer to do Y this way or that way?" produces social performance data dressed as preference data. The distinction is the difference between a research session that changes something and a research session that fills slides.

The structured interview guide problem.

Most interview guides are built backwards. A PM or researcher decides in advance what they want to learn, writes questions that address those topics, and conducts the interview by working through the list. This approach is systematic, which makes it feel rigorous. It is also the mechanism by which your pre-existing assumptions get smuggled into your data.

When you decide in advance what you want to learn, you are also deciding in advance what is worth learning. Anything that doesn't map onto your predefined topics gets filtered out — not maliciously, but because the interview guide gives you something to move toward when the conversation goes somewhere unexpected. The researcher redirects back to the guide. The unexpected thing disappears. The guide produces exactly the findings it was designed to surface.

This happens in the most earnest research programmes run by the most technically capable researchers. The problem is not effort or intention. It is structure. A question guide that says "ask about onboarding experience, ask about daily workflow, ask about feature X" will produce findings about onboarding experience, daily workflow, and feature X — and will miss entirely the thing the user actually cares about, which was not on the list because the PM didn't know to put it there. If you already knew what mattered, you wouldn't need the research.

The fix is not to abandon structure entirely. It is to front-load the interview with open-ended behaviour questions before introducing any of your hypotheses. "Tell me about the last time you had to do this task from start to finish" before any mention of your product or your features. The user's unprompted account of what happened — the sequence, the tools used, the points of friction named in their language, not yours — is the most valuable part of the interview. Most guided interviews never get there because the guide starts in the middle.

Why demos invalidate your session.

Showing the product during a research session is the single most efficient way to invalidate everything that comes after it. Once a user has seen your product, every response they give is a response to your product — not to their actual experience, their actual workflow, or their actual pain. You have replaced reality with your prototype and asked them to evaluate the prototype. That is a usability test. It is not user research.

Usability tests have their place. They answer "can users do X with the product we have built?" They do not answer "is X the right thing to have built?" If you want to know the second question — and the second question is the one that changes strategy — you cannot show the product. You must ask about the world as it existed before your product entered it.

A common version: the interviewer introduces the product in the third minute of the interview, spends twenty minutes on a demo and feedback, and then writes up the session under "user research" with findings about user needs. The findings are accurate observations of user responses to a specific prototype at a specific moment. They are not user needs. They are product reactions. Using them to make strategic decisions is like asking someone what they think of your restaurant after they've eaten the food — it tells you something about the meal, nothing about whether they were hungry for it.

The participants nobody is recruiting.

User research recruits the users who are easy to reach. Active users, recent signups, engaged customers, people who already like the product enough to respond to an interview request. This is a sample that is systematically biased in one direction: survival bias. You are talking to the people who made it through your funnel. You are not talking to the people who left, the people who tried and gave up, the people who looked at the product and clicked away in the first thirty seconds, or — most importantly — the people who have the problem your product is meant to solve but have never heard of you.

The people who have the problem but are not your users are the most valuable research subjects in almost every product context. They tell you about the market, not the product. They describe the problem in the language they use to describe it to each other, not the language your product has taught your users to adopt. They are managing the problem with competitors or workarounds, which tells you what they actually value and what they will tolerate. And they have not been educated by your product, which means their mental model of the problem is uncontaminated by your framing of the solution.

These people are harder to recruit. They don't respond to in-app prompts because they're not in the app. You have to find them through communities, referrals, job postings (people hiring for the role that has the problem), or cold outreach. The effort is real. The delta in insight quality is also real. The best product insights tend to come from research sessions with people who were currently managing a problem with an Excel spreadsheet or a WhatsApp group — not from sessions with power users of the product that was ostensibly trying to solve it.

What actually produces signal.

The sessions that produce genuine insight share a pattern. The interviewer begins with no agenda beyond understanding a specific behaviour or workflow. They ask the user to walk them through a recent, specific instance. They follow the user's language, not their own. They ask what happened next. They ask who else was involved. They ask what the user did when something went wrong. They resist the urge to introduce hypotheses until they have a complete picture of the current behaviour.

Insight arrives in the gap between what users say they do and what they describe doing. A user who says "oh, I just track it in our system" and then describes a process that involves three different tools, a manual export on Fridays, and a weekly standup to reconcile discrepancies — that gap is where the product lives. The user doesn't see it as a problem. They've normalised it. The researcher's job is to notice the gap and probe it: "When you say you just export it manually, how long does that take each week? What happens if someone edits the source after you've exported?" The answers to those questions are the brief.

This kind of research is slower, messier, and harder to summarise in a forty-two-slide deck. It produces findings that are more ambiguous, less statistically satisfying, and much more likely to be surprising. That surprise is the point. If the research doesn't produce at least one thing that challenges an assumption the team held before the session started, the research hasn't worked. Done well, user research should make product decisions harder in the short term and better in the long term — because you are building for the actual problem rather than the one your team imagined into existence at the whiteboard.

• 01
  Ask about past behaviour, never future preferences. "Would you use this?" is a social transaction, not data. "Walk me through the last time you did this task" produces observable behaviour you can reason from. Past behaviour is the only material that produces signal. Future preference is the material that produces decks everyone agrees with and nobody uses.
• 02
  Do not show the product in a discovery session. Showing your product converts a research session into a usability test. The findings become responses to your prototype, not accounts of user reality. If you want to know what to build, keep the product out of the room until after the user has fully described their current world.
• 03
  Start interviews with no agenda except the behaviour you want to understand. A question guide that covers your hypotheses ensures your hypotheses are confirmed. Start with "tell me what happened the last time you had to do X" and follow the user's account wherever it goes before introducing any of your framings. The unexpected direction is usually where the insight is.
• 04
  Recruit the people who left or never arrived. Active users tell you about the product. People who tried and quit, or who have the problem but don't use you, tell you about the market. The second group is harder to recruit and far more valuable. Wherever your best users are not — that's where you need to look for the insight that changes strategy.
• 05
  Listen for the gap between what users say they do and what they describe doing. The user who says "oh, we just track it in our system" and then describes a four-step workaround involving three tools is showing you where the product lives. Follow the gap. Ask how long it takes. Ask what happens when it breaks. The normalised workaround is almost always the real product opportunity.
• 06
  If the research doesn't surprise you, it didn't work. Research that confirms every prior assumption is confirmation bias with a budget. Good research produces at least one finding that challenges something the team believed before the sessions started. If you can't name that finding at the readout, the sessions were structured to avoid it — and you should be suspicious of every recommendation that follows.

Frequently asked questions.