‘Discount’ user testing under fire
“It has become almost a truism that tests involving five different users will reveal more than 80% of problems with a design and that the law of diminishing returns means tests with further users reveal less and less useful information. …
However, weaknesses in this approach were exposed along two dimensions by conference speakers. First, Dye, group manager MS Marketing Intelligence, warned that analyzing individual features of a complex products may improve aspects locally while ignoring the needs of the user. He spoke of the obstacles to better design: the challenge of understanding human activity; software technologies being difficult to build and our poor knowledge of work requirements and goals. ”
This is an important article. We need to reshape our attitudes toward discount usability. Not so long ago I was lucky enough to do discount usability and conduct a heuristic evaluation, and the HE was much more effective at revealing a breadth of issues, while the guerilla testing could only scratch at the surface of the product’s problems.
That said, having the developers sit in on the sessions made a huge difference in getting changes made.
Because it is seen as less formal, discount usability is often plagued with problems of carelessness and inexperience, including
- improperly moderated/designed
- hastily held, thus not covering the entire system
- often held by designers or done internally, and buffeted by the winds of internal politics
- testing small section of a product leads to ignoring the big-picture problems to concentrate on GUI and labeling tweaking
- five users is not the same as five good users– recruiting only five does not allow for a bad recruit or a no-show
When it is done right, it is more effective at flushing out design disasters during the design than evaluating the entire system pre-release. It can be a swift way to shake designers out of design mode, and reveal usage-related problems. it can reveal mental models, show design advantages and disadvantages, and is thus great when snuck in before conducting a redesign. But used before shipping a hefty complex piece of software? or before launching a thousand page site with rich functionality? Here you want to rigors of formal usability to assure your company’s reputation won’t go to sea when you ship.
in my personal opinion, discount usability when done correctly is an excellent design tool and a poor evaluation tool.
I’d like to hear your thoughts on this matter… what problems have you seen with discount usability testing? What wins has it given you?