Customer Validation · Customer development

What is an ideal time and the metrics that should "validate" a startup idea?

Ryan Green FX Sales and Trading at PNC

November 6th, 2014

I currently have a landing page up to validate a current hypothesis to see if people would actually be interested in it. I am just trying to understand, when is it considered validated, how long should I test the idea, and what are the most important metrics?
A great idea is 1% of the work. Execution is the other 99%. In this course, we’ll teach you how to conduct market analysis, create an MVP and pivot (if needed), launch your business, survey customers, iterate your product/service based on feedback, and gain traction quickly.

Sam McAfee Building Popup Incubators for Corporate Innovation Programs

November 10th, 2014

@Chris, that sounds really interesting. I am going to check it out, and pass it around to the other folks here at Neo.

@Ryan, in my experience, you really can't know whether you have product-market fit until pretty far along. A landing page is not going to do it. That only tests the value proposition and the channel. There is no ONE hypothesis in your model, but several, and they will continue to come up as you remove other ones. If you look at your Business Model Canvas, which I know you have because you're a smart guy :) , you need to construct experiments for each of the boxes, and validate your assumptions therein. Do them in the order of highest risk, I'd suggest.

At Neo, we tend to talk about this in terms of a continuum of gradually reducing uncertainty, through gradually increasing fidelity of experiments.

2a0ab1fc542b8ac1e2c8a3f918fdb9ede0331e20.png
My default path is typically to validate the customer segment by drawing up some personas (just lightweight sketches and demographics) and seeing if those people exist. If you can establish that, it's time to move on to problem interviews, and then paper prototypes or sketches of your solution. Only at that point, is it time to build a landing page and start testing your value proposition in different channels.

But that's just the beginning. I'd be careful talking about product-market fit until your numbers actually balance out. In other words, you've built a financial model of your product, and you have a pretty clear idea of how many customers you need in order to break even at least. That's based on your funnel metrics (acquisition, activation, retention, referral, and revenue). You need metrics in all those categories to even come close to a clear idea. This would probably be several months into building and delivering something people want. So, you see, it's not a clear threshold at all. It's a gradual reduction of risk.

I have seen startups that are 3 years old, have customers, maybe a couple rounds of funding, and it's not entirely clear that they've nailed it yet. It really depends on industry, but if you look at investors in A and B rounds, they are interested in knowing that if they put money into your company, it's going to accelerate what you already have going on, and not just get burned up. They certainly aren't going to do that if the numbers don't work out (unless there is some insanely compelling narrative that they can't resist; but I wouldn't rely on that).


Chris Sorensen Startup Executive / Mentor, Expert in Innovation and Product-Market Fit, MBA, MS

November 6th, 2014

Hi Ryan,

Great question.

I believe that we are starting a new "golden age" of global entrepreneurship (and I'm not the only one).

But has as we saw during the last  dot com bubble in 2000, as the number of startups increased the success/failure rate stayed fairly constant.

This may suggest that there isn't an absolute limit on the number of "good ideas" that can succeed at any one time as some people have suggested, but rather that we need a better process to reduce the number failures.

Initiatives like Customer Discovery and the "Lean Startup" movements are making some progress in improving the process. But so far as I know, no one has proposed a metric that evaluates a company's chances for success.

So we created one.

Its called "Quantitative Product Market Fit" (see www.Q-PMF.com)

Marc Andreessen wrote a blog post in which he posited that "Product Market Fit was the only thing that mattered...what else could it be?"

We have taken the next step and believe that what it could be is not just a product's level of fitness, but the degree of fitness (or value) that it provides over other competing options.

We call this metric the Value Delta (or "Delta-V" for short)

We believe that Delta-V explains ALL other business performance metrics including market share, loyalty, cost of customer acquisition, net promoter score, etc.
 
Please visit our blog, or see the slide deck on my Linkedin profile for more information.

Best of luck for great success!

Chris Sorensen




  

Nathan Terrazas

November 9th, 2014

Ryan- 

It definitely depends on what the hypothesis is....

BUT. A helpful metric in this instance is to figure out your total served market (how many people within your total available market could be compelled to buy) relative to your landing page metrics. I'm assuming you collected email addresses on the landing page. 

Keep this in mind- landing pages often don't tell the whole story. One quality conversation with a prospect is worth more than 100 email addresses. If you're receiving a steady flow of email addresses from a landing page, thats a signal that your idea is at least interesting enough to others that you can contact them to get high quality feedback from a conversation. 

Good Luck!

Chris Carruth VP/Director. Strategy | Business Development | Operations | Product | Solutions

November 6th, 2014

@Chris - that's a great idea, will be interesting to see how accurate it is.

@Ryan - success varies by business objective. A failure for one could be considered a success for another. For example, most consumer products have a failure rate of 70% to 90%, meaning a chance of succeeding 10% to 30% of the time. In video gaming, the published success rate across all genres, i.e., becoming profitable, is about 4%. In grocery, private label  "purchase intent" scores have to be above 30% while for national brands they look for 70% or more. Which one will be successful - all could be, for different reasons by different measures in different industries.

Decide what the goal of the company is, define how a successful product supports that goal (top line revenues, net profit, units shipped, user base expanded, etc) and then breakdown that success into measurements that make sense AND are key to the company's growth. You might be able to measure product success a dozen ways but probably only 2-4 have a REAL impact on company objectives..and its' success.

C

Joseph Hsieh Growth Marketing & User Acquisition Specialist

November 10th, 2014

Some of these answers above are really amazing.  Thanks for sharing those.

For the startups I've been involved in the past, we moved from the weaker "LP email optin metrics" from to strong commercial intent metrics.  Surely, people will provide an email if they are just curious.  Depending on your product, you can do qualitative email follows (with a survey) or simply call them and ask them for the check!  

"If I were to build this, would you pay $X/month?"  (or whatever your pricing you were planning - which is a whole other set of discovery you would want to know.)  

If not, ask them why or what would be valuable for them to pull out their credit card.  Of course, you may get "faster horses" (reference to Henry Ford about asking customers what they want) type of answers, but at least you get a sense of the commercial value of your concept.