TL;DR: Pilot interviews are not polite feedback calls. They are manual, unscalable sessions to gather raw evidence of whether your product actually creates value. Do not ask users if they like the product. Ask them to walk through exactly what they did, where their workflow broke, and what would happen if the product disappeared tomorrow.
Many founders leave a pilot call feeling relieved because the customer said the product was "great." Then they look at the data. The customer has not logged in for four days, has not invited their team, and quietly went back to using a spreadsheet.
This happens when you treat customer insight interviews as a polite feedback ritual. You avoid asking uncomfortable evidence questions during the pilot. You collect polite praise, assume you are on the right track, and keep building from your own assumptions.
Customer insight interview questions are specific, behavior-focused prompts. Use them during early product usage to uncover hidden onboarding friction, whether the user actually gets value, and silent churn risks. If you do not collect real evidence of past behavior, you do not know the customer. If you do not test their willingness to pay, you have not produced real value. A pilot interview is not a search for compliments. It is a search for what actually happened.
Why Scalable Insight Gathering Fails Early Pilots
Founders often watch the successful onboarding flows of big software companies and think their early pilot needs to be scalable. They use automated emails and opinion polls to check in.
Do not overcomplicate this stage. Early pilot work trades scale for direct communication and deep context. Sometimes onboarding should remain manual and unscalable for a long time — as seen in Superhuman's early growth strategy. The goal is not to impress users with a slick process. The goal is to build a sharp mental model of the customer.
When you hide behind scalable scripts or ask hypothetical questions like "would you use this?", you get polite lies. You must sit with your first alpha customers, understand their team context, and watch where your product collides with their real workflow.
The Framework: Surfacing Hidden Objections
Getting useful answers requires a setup where feedback is expected. Establish this dynamic early with a Design Partner MOU (found in our customer discovery kit) so you have the right to ask hard questions during your startup pilot program.
Pilot customers will rarely volunteer what is blocking adoption or renewal. If you ask, "Are there any blockers?", they will say they have just been busy. You have to surface those objections yourself, because unspoken objections decide the outcome after the call ends.
Instead of asking for their opinion, use this framework to capture raw evidence:
Find the expectation gap: Ask exactly what problem they expected solved before the pilot started.
Force the comparison: Ask what they compare the product against and what else they tried.
Trace the failure: Ask why those alternatives failed or were never attempted.
Surface the doubt: Ask what doubts remain about using your product going forward.
Capture these answers as raw evidence first. Do not try to immediately group them into themes or product recommendations. This approach aligns with the core principle of The Mom Test, which teaches founders to ask about actual behavior rather than future promises.
12 Customer Insight Interview Questions for Pilots
Here are specific starter questions to ask during active pilots. They are designed to reveal onboarding friction, value, and potential churn risks.
1. Questions to Expose Onboarding Friction
Do not ask what they think of the dashboard. Use a customer discovery questions script that focuses entirely on past behavior.
"Walk me through the exact steps you took the last time you tried to use this in your weekly workflow."
"When you hit the data export step, what did you actually do next?"
"Who else on your team needed to be involved, and what did they say when you showed it to them?"
"At what specific moment did you decide to go back to your old method?"
2. Questions to Test if They Actually Get Value
You need to know if the product is solving an urgent pain or fulfilling a real job to be done.
"If we turned this off tomorrow, what exactly would break in your workflow?"
"What would you go back to using if this disappeared?"
"Can you show me the last outcome you produced using our product?"
"How long did it take you to get to that result, compared to your previous tool?"
3. Questions to Surface Churn Risk and Willingness to Pay
"Customer said they would upgrade" is not a reliable signal. You need to know if they have the authority and the actual intent. If answers reveal that pilot users do not have the pain or budget you assumed, revisit the core questions to ask your target market to see if you targeted the wrong segment.
"Whose budget would this come from, and what do they think of the category?"
"What would need to be true for you to pay for this next month?"
"What internal pushback do you expect if you ask to buy this?"
"When you think about paying for this, what other tool's budget are you comparing it to?"
Signal Check Table
Use this compact guide to match pilot behaviors with the right questions.
Pilot signal | What to ask | What the answer reveals |
|---|---|---|
Logged in once, then stopped | "At what specific moment did you decide to go back to your old method?" | Hidden friction or missing core feature. |
Praising it, but using it rarely | "Can you show me the last outcome you produced using our product?" | Whether the problem is actually painful enough to solve. |
Asking for a free extension | "What would need to be true for you to pay for this next month?" | Lack of authority, lack of budget, or lack of value. |
FAQ
Can I just ask pilot users what they think and whether they like it?
No. These questions produce polite lies and hypotheticals. In active pilots, ask about what the customer actually did, where onboarding broke, when they found value, and why they behaved that way. Your job is not to collect opinions about yourself. It is to build a mental model from past behavior and real value, even if that means running manual, unscalable interviews with your first alpha customers.
When should I run pilot interviews?
Run them at specific friction points, not just at the end. Check in after they complete onboarding, after they attempt their first core task, and two weeks before the pilot ends.
How many pilot users should I interview?
At the early stage, interview every single active pilot user. You are looking for deep context and workflow understanding, not statistical significance.
What should I do if users say they like it but do not use it?
Stop asking if they like it. Ask them to show you how they currently solve the problem. If they have not taken any action to solve it using their old tools either, the pain is likely not acute enough to support a business.
