Usability testing

Tools and Best Practices for User Testing?

Lucia Guh-Siesel CEO & Founder, Bandalou

July 12th, 2014

We are about ready to begin user testing with approx 40-50 friends & family.  What are some good tools and best practices for tracking user comments and feedback?  Any good inexpensive options that you've used or heard about in the past?  Thanks!

Eric Sexton Game Desginer at Crate Entertainment

July 12th, 2014

Great question.  I am also currently starting testing and would love to know the same suggestions.

I am currently using Trello.com to organize and manage my team tasks.  My test group is still really small so I am using a private Facebook group, but once we have more than 10 testers this is going to be unmanageable.

Dimitry Rotstein Founder at Miranor

July 12th, 2014

With 40-50 friends/family the best (and the cheapest) option is e-mail. 40-50 emails is well within the reasonable limits for one person, and it doesn't introduce any boxing or added learning on user feedback (any dedicated tool probably does). The best practice, by the way, is to ask users open questions (what do you think? what's missing? what's redundant? What was the biggest obstacle in using the product? If you could change anything, what would it be? etc), not multiple choice or yes/no, nothing that can limit users' creativity.

Bob Binder Member of Technical Staff at Software Engineering Institute | Carnegie Mellon University

July 12th, 2014

Basecamp provides simple issue tracking and more. With a 60 day free trial, you can probably support your beta test at no cost. https://basecamp.com/

Don Daglow 3-Time Inc. 500 CEO, Technical Emmy® Award, International Speaker, Advisor at Founders Space accelerator

July 12th, 2014

I may be answering a different question than the one you asked, but here goes.

For tracking issues I agree that Basecamp or Trello should work well.

I'm assuming that you're going to get two kinds of reports: functionality and bugs.  They represent two totally different response cycles.

Bugs are objective: the app crashed.  The page never displayed.  I pressed the button to get red marbles and I got blue Legos.  The screen took forever to load.  I hit the back arrow but nothing happened.

There are specialized bug tracking data bases, but if you're conscientious about tracking the bugs you can do almost as well with a spreadsheet.  
  • Make sure all your friends and family testers give you all the necessary information for each bug: exactly what they saw that did not look or feel right, what they were doing when it happened, what screen was displayed, what they had been doing before they did the action that displayed a bug, if and how they were able to recover and continue (if it's a navigation bug etc.)  Be sure to get the device and browser they were using and the version of that browser, since Safari, IE, Chrome etc. will react differently with different version numbers.
    • Some bugs are easy: there's a missing image on a page, and it can be reproduced every time just by going there.  On those all the other data can usually be skipped.
    • Some bugs, like memory leaks, are hard to reproduce, because it's a long chain of activities that eventually overflows some bucket and produces bad results.  Nevertheless, still get all the details, because what looks like random crashes to you will eventually reveal a pattern to the engineer.
  • Give every bug its own identifying number in the spreadsheet, and don't re-use numbers after the bug is fixed.  This gives you a trail of issues and sometimes allows engineers to go back and see where a fix for one bug created another.  It also lets you track the ratio between new bugs appearing and old ones being closed, which allows you to see your progress on producing stable code.
  • Mark each bug as "Open", "Claimed Fix" (we think we fixed it but a test pass has not yet verified that the fix works correctly, and "Closed", for when you can clearly see it's fixed.
  • Give each bug a priority of A-D (or 1-4 if you prefer), where the highest priority are crashes, next are obvious problems anyone would notice, third are problems you notice and want to fix but don't sabotage functionality, and the last are "nice to fix."
  • Copy over into the spreadsheet ALL the text about the bug you get from the user.  This may produce a messy looking spreadsheet with big boxes, but it's the only way to actually help your engineers.
  • Attack bugs in priority order
  • Sometimes engineers will have fast, easy fixes they do as a break between hard ones.  This is good with A's and B's, but there's a danger that doing this with C's and D's can delay getting the more important stuff done.
  • Review the top of the bug list with your team every day as you approach a major release (or just use Agile dev methods) and make sure it's clear who is taking on which bugs that day and what happened with the ones they took on yesterday.
  • Continually update the spreadsheet (at least once a day) with new bugs and turning opens into Claim Fixed and verifying fixes and closing bugs.  This gives the team a sense of momentum as they can see their progress.

That's all the objective stuff.  For the subjective feedback and even the data-driven analysis feedback, it isn't such an obvious "here's a problem and we know how to solve it."  You can use a spreadsheet, Basecamp, Trello etc. to catalog and organize these issues.  Some will be troubling and no clear good answer is present.  Some will be "Do you prefer green or purple?"  Some will be at the highest strategic level, "Is it time to pivot and focus on x instead of y?"

Whatever software you use, the key is to not confuse the objective bugs with the subjective features and functionality.  The team may be making changes on many issues from each column at the same time, but the process for deciding what gets done and in what order is completely different.

Hope this is of help to you, and that I didn't cover a lot of territory you already know well.


Don Daglow







Laurence Briggs

July 13th, 2014

At PrevaLeaf e are currently consumer testing five natural products in the feminine health space.
Survey monkey has been perfect for us.
Simple to create a survey for each product and then we ask approximately ten questions per product.
It is also easy to set up and easy to modify.
Their model is freemium. If you want to see it in operation check out 'feedback survey' on www.prevaleaf.com
The problem we found was in the conversion i,e, getting people to take the time to go online and fill out the survey. Many people told us the products worked fabulously and were really happy to have them but making the effort to go online......well we had to figure that out. 
Starbucks cards as rewards did not work. We found the solution rapidly by having (paying) a nurse to call each person 15 days after they had received the sample.
I hope that helps

Lawrence Lerner Digitalization and Transformation Coach

July 13th, 2014

Great questions and @Don makes some good points. There are number of ways you can track what your users are reporting.

Online survey tools such as SurveyMonkey (Free for small use) or LimeSurvey (Open Source and always free) are other tools you can use to capture feedback. 

There are a number of critical steps you want to capture that improve your product and not just identify bugs. There are some techniques that can be used with Risk Based Testing to help identify what is most impactful from a business and technical aspect.

I've run some very large (>2500 FTE) testing businesses. Feel free to ping me offlist and I am happy to share some best practices  

Karl Schulmeisters Founder ExStreamVR

July 29th, 2014

Jira - https://www.atlassian.com/software/jira?utm_source=twit&utm_medium=radio&utm_campaign=twit  Provides a free 30 day trial if you follow the above link.  They can be deployed as a full blown project/issues management solution, or for small projects.

I'm not sure why you think that with 40-50 users you will be overwhelmed using something like a private forum on FB.   You are going to have to triage and track each bug or suggestion anyway.  So adding the data entry time, which consists off two "Cut and Paste" steps (the author and then the text)   really doesn't add that much time

Scott Tolstoy Teakah founder

July 29th, 2014

For Customer Service and interaction with clients I would recommend http://www.desk.com/
it's cloud base software and have free trial 

Julie Blitzer

July 30th, 2014

I did a class for General Assembly last year on this exact subject - watch free as part of their two week trial and it should answer nearly all of the questions you asked in your post. Good luck! https://generalassemb.ly/online/videos/ux-research-make-better-products-with-usability-testing