Jane Morgan gives us some practical hints and tips about qualitative usability testing.
Finding things that are not good, but good to know; that is the core outcome of usability testing. Other research methods are superior for uncovering unmet customer needs or sizing the market for your solutions. Usabiity testing focuses on those who touch your product providing valuable guidance on how to improve.
This face to face method (which can also be doing using screen sharing technology such as GoToMEeting or Skype), provides rich input for product development. It assumes a website or software interface and the concept can be adapted for other product types.
This practical guide is for those who want to know:
- How can I do face to face qualitiative usability well?
- Does the software / website functionality solve the customer need elegantly?
- If not, where is the customer getting stuck?
Usability Testing – The Checklist
- Usability Script
- Usability Testers
- Your Observers
- User testing observation spreadsheet Template
- Schedule
- Software to test
- Screen sharing tool (optional)
- Minimum of two laptops or PCs
- Testing location (potentialy online)
- Moderator’s Script
- A Moderator
When should you do usability testing?
Some say it is never too soon to start usability but frankly, I think, that insults teams. You will want to narrow in on the unknowns, not spend precious resources on confirming what the team already know.
Conduct usability testing when there’s time to implement changes, but after you’ve worked through common issues that impact users.
What to test?
Don’t test everything. Only test what you’re willing to change. Consider navigation, status, and actions. Ask yourself:
- Where do your development team want tested?
- Which implementations generated the most team discussion?
- What are the absolutely critical customer needs?
- Which features are unique or new to this offering?
Make the list short enough for a usability script. If it's too short either you are fooling yourself or….. you're fooling yourself!
Developing a usability script
The script outlines the flow, the process for your usability testing. The better the script, the more successfully it will fade into the background during testing. Some tips:
- State the customer problem/need (not how to complete the task)
- Sequence tasks logically (e.g., create a file before saving it)
- Don’t befuddle the tester/customer (use their language)
- Use large font and few words
- Number tasks so observers stay aligned
- Test the length
- Keep each session as short as possible, definitely under 45 minutes.
- Print one task per page to prevent testers from reading ahead
Special Note: if you are considering filing patents, you’ll also need testers to sign a non-disclosure agreement (NDA).
Usability Testing – The People
Group 1 – Usability testers
I call this first group of people "testers". They are testing, not being tested. You’re looking for honest, even unpleasant feedback from folks who represent your customer personas. No one on the development team counts. Not even the boss.
How to find usability testers?
Start early, and consider these ideas for finding usability testers:
- Existing customers
- Ask colleagues (e.g., if your target audience is parents, are there parent among other teams?)
- Work with supports and sales
- Work with marketing wh have exisiting customer communication methods
- Ask your friends, and friends of friends
- Add a sign-up form on your website
- When doing other research, ask if you may contact participants again
- Use professional recruiters
- Work with a research professional
Special note about post-sales support teams: There is nothing typical about the user experience of this group. With deep product knowledge, they don't talk to your average customer. Feedback from this group is often very precious, and seriously warped.
How many usability testers?
In Jane's experience, about 10 are required for confidence in this type of rich, qualitative feedback. With 10 testers, you’ll have more data than you can shake a stick at. Jakob Nielsen advocates doing small usability tests with around five testers.
Group 2 – Usability Moderator
The moderator chairs the session and is the only one who speaks with testers. A moderator either develops or destroys your usability testing. Team members typically self-select for (and against) this role; often project managers make a good choice. Alternatively using a professional can be very useful.
Key characteristics of a good moderator
A good moderator is someone who:
- Has experience speaking to customers
- Sets the stage for frank feedback
- Knows how and when to prod (e.g., "I see the mouse is not moving. What would you expect there?")
- Does not lead the witness
Group 3 – Usability Observers
Learning and capturing actionable information is the observer’s role. Each observer participates in at least three sessions to avoid being overly influenced by one strange usability test. One tester may glide effortlessly through a labyrinth, while another is left baffled.
If you can have someone participate in all sessions. Spending a few days observing is usually a lot more exciting than reading the report afterwards. Alternatively, have one person get intimidate with that cosy couch and read each report.
Where can you find observers?
Is it clear this group is your development team? It's a role that fits both interns and experts, the shy and the bubbly; everybody learns something.
User design and product management teams are usually first in the queue. Engineers often invent solutions as they observe. You’ll likely only have to persuade them the first time. Project sponsors and influenetial peers become project and customer advocates by participating.
You need 3 observers for each usability test. Number of testers x 3 = Number of observers
The Test Location
You’ll need screen sharing software which shows the tester’s desktop, full screen. (The image below suggests how to lay out the room.)
Some people suggest recording but I have yet to see a team use recordings; your memory, and colleagues memories work very well.
Data Collection
And what does "usable" mean, anyway?
The goals is to learn and understand the core answers to two main questions:
- Does the implementation solve the customer problem/task? Did the tester complete the task?
- What did it take for the customer to complete the task? This subjective question provides rich feedback. Typically, teams are surprised at what testers do.
Usability Observation Template
Collating free-form text kills enthusiasm for many a worthy projects. Your liberator? The spreadsheet.
Provide observers with a digital or paper spreadsheet. This is a team effort; notes are for sharing. With document sharing options to easily available though platforms such as Dropbox and Google Drive, the timetable, template, feedback, and analysis can reside in a single spreadsheet which all team members can access.
Download the JEM 9 User Testing Observation Spreadsheet Template free of charge now.
Usability Analysis
If you are strict about using the user testing observation spreadsheet, this process is straightforward:
- Without changing the formatting, copy all rows from your observers’ feedback into one spreadsheet (rows seven and below are in the example shown)
- Sort by task
- To locate the worst features, first add a subtotal for each task, then add up the total minutes
- Filter out completed tasks (those marked "yes")
- Decide on a course of action (next steps)
Start with the incomplete tasks that took the longest. You may decide to consider completed tasks later, but fry the biggest fish first. Now you’re back in regular development prioritization mode.
The insights gleaned from usability testing inspire teams and improve your solutions in ways you likely never imagined. Now, go make those products and website better!
About Jane Morgan
Having worked for 20 years with technology teams from Boston to Billund, Paris to Providence (RI), and Berlin to Bengaluru, Jane founded JEM 9 to assist organizations in understanding and reaching customers. At JEM 9 Marketing Consultancy – Helping You Understand And Reach Customers