Lean startup · Customer Validation

Success criteria when running a lean experiement for a paid service?

Claudia Goga Founder at CareToPets

February 18th, 2016

I was wondering if any of you have some recommendations for setting up a minimum success criteria when you're running a lean experiment?

I've read that collecting 50 email addresses is a strong sign of validation and 10 a weak sign.

But how about paid services? What would the numbers should look like? Also what if the service you are testing is something new to market, would the success criteria be different?

Would 2 customers that actually paid, considered a sign of validation when testing a subscription commerce service, for example?

Rob G

February 18th, 2016

i don't think the answer is so binary.  The answer depends on: 
1. what you are selling
2. to whom
3. for how much

For example if the product you are selling is a SaaS service to large/enterprise customers and 1,000 companies make up 80% of the target market and you charge $1M/year then 2 customers that fit the 'ideal customer' profile signing a letter of commitment/intent or committing resources for implementation would be pretty good initial validation, especially if you only met with the decision makers of 4 companies.  "success" RATE should also be a validation criteria - if you speak to/meet with 10 companies who fit the ideal prospect profile and 2 sign up that's pretty solid validation.  If you send 1,000 emails and get 2 "maybe" responses then you need to do more homework.  On the other hand if your service is one more home grocery delivery service or another Uber for home cooked meals or a consumer photo sharing service then i would want to see more like 50-100 'committed' customers out of perhaps 500 contacts.  This validation process - "product/market fit" process, is really for your own edification, not some investor pitch.  You should be your own harshest critic.  Keep validating the product/market fit until you KNOW you are on to something valuable.  You shouldn't need to convince yourself, the market should be convincing you. Talk to 10 prospects that fit your ideal customer profile. If 0-2 show sings of interest then you need to go back to the drawing board.  If 5-10 say "this sounds fantastic, when can i have it" and half of those make commitments (make a down payment, sign a letter of intent, commit internal resources to deployment, etc.) then that's a pretty good sign you are on to something.  

When you say "paid services" are you talking about SaaS services or live bodies doing work for a fee? 

Luke Szyrmer Forward-Thinking, Creative Software Product Manager and Author

February 18th, 2016


What matters is the number of conversions relative to the traffic that hits the page, or the conversion rate. 50 or 10 may be a lot or a little. 

A paid conversion hold much more weight than just an email. In fact, if you have someone paying you already, that means you have problem-solution fit and you have strangers willing to pay you to solve their problem. Which is a very big step forwards. 

Sam McAfee Building better technology leaders and teams

February 18th, 2016

Here are some resources on this. Make sure you're solving a problem for a market segment that actually exists. Then make sure you're running the right kind of experiment for what you're trying to learn. The success metrics you use should be based on whether you're trying to validate the customer segment, the problem, the solution/service, the market size, or something else. You need to make sure you've sketched out your business model first, or you won't know what to test.

(Disclaimer: the first link is by me). :-)

Dimitry Rotstein Founder at Miranor

February 18th, 2016

It depends on many factors - hard to put it in universally applicable numbers.
That being said, having even a single paying customer, unless it's a close relative/friend, is an achievement in itself. Still, personally, I'd feel better having at least three paying customers. One might be a fluke, two - a coincidence, but three and more - that's either a beginning of a trend or a lucky strike (both are good :-)
Unless, of course, your product is Enterprise Software, in which case having even a single paying customer is one hell of a start - shuts up all the "unbelievers" like that.
Of course, 3 or 10 or 50 is just a start. The real test of a paid business model is to estimate the CAC (customer acquisition cost) and the average LTV (life-time value) of these customers. LTV should be at least 3 times the CAC, or you won't be profitable. It's tricky, since there are many hidden costs that are hard to see, but this calculation is imperative.

Rob G

February 18th, 2016

sam makes some good points... a reminder that i shouldn't assume that the poster knows what s/he is actually testing for, that they have already identified and validated a real problem and already defined the ideal prospect and are thus solving for "is this a good solution to the validated problem for this defined market...".