Design · Analytics

How soon is too soon to make data driven design decisions?

Michael Evans Filmmaker / Creative Director

February 18th, 2015

Should user behavior dictate design or design dictate user behavior?

Growth-hacking isn’t about quick wins and shortcuts, although they exist. In this course, we’ll cover the six-step growth hacking framework, how to measure user retention for your business, how to increase engagement and retention, and a bunch of case studies.

Jared Hardy Founding Director at Data Roads Foundation

February 18th, 2015

It's always going to be too soon to make any decisions based on bad or unknown quality data. The first step is always to look at all data ingress paths, including full step by step audits if necessary. Third party data without peer review of full ingress channels and analysis is all basically useless. As with push polling, design always dictates the limits of user behavior, so expand the design space in order to expand the range of user behavior testing. The easiest way to avoid your own and others' preconceived biases is to randomize key elements of the customer interface, track all interaction against key performance goal outcomes (usually culminating in a purchase or given dollar amount), and then map back from goal completions to consistent interface elements and placement among the random set. Useful features emerge over time, never on the first time out.

You should never consider any design "final" until you at least A|B test it against alternatives. If you can afford it, take at least two completely different designs from non-colluding sources given the same feature requirements and performance goals, and A|B test them against those goals until there is a clear (statistically significant) winner. Iterate A|B tests on the winner with random details taken from the loser, to narrow down which individual features actually impact the outcome. Never settle on any design element until it is vetted against competition, and evaluated by the actions of real paying customers. Paid "experts" and "focus groups" are useless by comparison.

Steve Jobs was right that customers do not know how to express exactly what they want, but he was wrong in that they do always know when they have found exactly what they want. The key is to know how to listen, by allowing them to express their desires through actions rather than words. Never survey what you can observe instead.

Joe Emison Chief Information Officer at Xceligent

February 18th, 2015

You should make data-driven decisions as soon as you have enough good data.

There are three elements to having enough good data: (1) accurate data, (2) on the right things, and (3) enough of it for it to be significant.

Accurate data means that you are measuring what you intend to measure. There are plenty of people who try to gather data by asking questions that influence the answer (e.g. "push polls")--and, in general, asking people direct questions and relying upon their answers is a bad idea. People lie to others to be nice (especially about their business ideas), and people are notoriously bad predictors of their own behavior. Your best bet is to measure what people are doing, not what they're saying.

Data on the right things means that there's a useful relationship between the data and what you want to do with it. For example, a common mistake I see is when enterpreneurs assume that a proclivity to purchase one item means a proclivity to purchase another. Or, in A/B testing, assuming that the reason why "A" won over "B" was because of a particular element of A, even if more things differed between the "A" and "B" experiments than that one thing.

Significant data usually means statisticallysignificant data, but doesn't necessarily have to--it can just be enough data to be compelling from an objective/reasonable-man standard. One problem with statistically-significant data is that one out of twenty garbage experiments is likely to be statistically significant, because we usually measure at the p=0.05 level (see You just want to avoid sample sizes that are too small, bad selection biases, etc.

But if all of the above are true, you should be making data-driven decisions. People are bad about letting their personal biases and "religious" perspectives (see get in the way of making good decisions. Data is an excellent antidote.  And given the choice between, "I am a Steve Jobsian genius, therefore my arbitrary feelings will lead to better decisions than data" and "I am following the data", the latter gives reliably better results.

Karl Schulmeisters CTO ClearRoadmap

February 18th, 2015

Depends Michael.
Usually its a blend  - where you design for an expected use case and behavior - but then involve users early enough in the process to understand how they will choose to use it.   The best example I can give you is how a custom architect works:

They get a design program from the client (ie a functional spec)
They then produce a design
The client gives them feedback on the design
The revised design gets built
Then the Client lives in the house and uses the spaces as THEY see fit.

Much has been made of Steve Jobs' and Jonathan Ives' singular visions for the iPod and iPhone - but its important to recognize that NEITHER product was revolutionary.  Each was actually an evolutionary design product vs. what was in the marketplace.   (heck that's even true of the original Mac UI and Mac Personal Computer).

There were plenty of MP3 players in the market before the iPod.  So Ives and Jobs could - and did - study how users used them, what worked, what didn't, what users liked, what users were trying to do but could not.

ibid iPhone.

So in so many ways, both for the iPod and iPhone,  some of the most iconic "revolutionary" products out there - there was no revolution.  USERs dictated the design and they were evolutionary rather than revolutionary designs

As Dimitrios pointed out,   the designs Ives and Jobs came up with resonated emotionally.   but that emotional response was the USER dictating how things should be - through his/her emotions

Yoli Chisholm Founder at WELL -

February 18th, 2015

Both... I echo the sentiments of both the gentlemen above and I'd add that one of the ways I practically apply this is as follows:

Let's say you are building a new app.... and you have some competitors in the space or an incumbent status quo you are looking to watch and record user behavior with the competitor or incumbent apps to discern what they do well and where the design flaws emerge as opportunities for you.

Then you let your design dictate the better experience which in some cases dictates a new behavior that if you are successful in your design the user cant imagine what they did without.

Like imagine a world without "cut and paste" :) or bluetooth or selfie mode, swiping all of which required new behavior...not great analogies for your question...but essentially in each case at some point someone observed incumbent behavior/design and designed new experiences that dictated new behavior....the answer I think is both

Stuart Frederich-Smith Director of Product at PlanGrid

February 18th, 2015

In my experience, data always helps inform the conversation, even if it is not statistically significant, and sometimes trends are obvious even with a relatively small sample. Conversion points articulate this well. If you've got a registration or lead generation flow it is an area you will probably focus on quite a bit. It is a good place to experiment early because user action will correct any internal biases you may have going into it. Time and again I've launched a particular design that I was fond of only to find that user's didn't agree. It takes time and iteration to find the right pattern, and instrumenting your designs such that this feedback is possible is tremendously helpful at any stage of the process. 

The other thing I've learned the hard way is that you should always preserve a backwards path when introducing new designs and keep as much control over the user flow in the hands of business users as possible. This allows performance experimentation to take place outside the dev sprint and a quick turnaround if things don't go according to plan.

kiranaimhigh kota Project Engineer at Unicorn Industries

February 18th, 2015

Hi Michael,

To my understanding, anything new invented all throughout the history was to make Human's Life easy and better. Human's in your case is the User.

Your Question is similar to two sides of the same Coin.

a) Design Dictates the User Behavior

   This happens when you have invented something which has never existed before. In this case you can dictate the "User" to some extent or over a period of time because this is when the people start using your product. After a period of time, use start bring in new ideas in the product, this is where we need to look into the other side of the coin.

b) User Dictates the Design

Design is always a set of ideas, when implemented makes products more user friendly for the "User" to use them.

Earlier, we used to have computers which were so big and over a period of time, the demands of the "Users" have increased which led to the innovation of Laptop and now smartphones.

That's why it’s always said that the CEO of any company need to have those futuristic thoughts or ideas to make products as user friendly as possible.

So, I think it depends upon whether you are an inventor or an implementer of an existing invention.

If in case you have queries, don't hesitate to contact me back on my email Id: