Researching Search

more outtakes from the 2nd edition of Blueprints, most likely destined for the long anticipated yet never begun book on search. Please Note: this has not been edited in any way

The speed of search causes other problems for teams new to it. Normally, to figure out how well your design is working, you might do a bit of usability testing. But that doesn’t work well for search.

Traditional research mars results

If you wanted to see how a hummingbirds fly, you’d have a hard time. They move so fast they become a blur. But if you asked them to slow down, they wouldn’t stay aloft and you couldn’t learn anything. .

When you are watching people try to complete a task in usability testing, you often use “think aloud” protocol, in which the user speaks their thoughts as they work on their task. But search is so fast, that if the user is forced to think, they slow down too much and behave unnaturally, thus creating false results.

With search, the physical actions are few… look, type, click. What you really want to know about is hidden, even from the person searching. Search requires different research methods

A few methods I’ve found useful:
Eyetracking: With other types of products, eyetracking is often a useless novelty. The data rarely tells you that much you can do anything about, since users consume multiple pages as they accomplish tasks. But because of Search’s nature… a single page, with endless but constrained variations… eyetracking can teach you quite a bit about the mental process as revealed in the way a searcher scans a page.

For example, on a results page, the searcher always glances at the query box first. When asked why, the answers vary… “I wanted to see if I misspelled it”, ” I wanted to see if the search engine got it right” … searchers often don’t really know their own minds. The important thing was they look there. That allows us to return to our rule, “every pixel has a job to do” and decide that yes, you need to show the query in the search box on the results page, and no, you do not need a label like “You searched for X.”

And that hotspot on the far right in the screenshot I’ve included? That’s the user actually seeing an ad that uses the keyword they searched for. Proving ads are seen when they are relevant. Imagine that.

Bucket testing: Also known as A-B testing. In this scenario you release different variations of the same design to a portion of your users, to see what is the most effective. While this is invaluable in understanding what works and what doesn’t in search, it’s also quite difficult to get trustworthy results.

A good example of a A-B test candidate: you don’t change anything else, but the color of the links.

 

To get good data from bucket testing, you need

  • Enough visitors in each group for good data
  • To run the test long enough to remove behavioral changes you see on weekends and certain week days. That means 7 days minimum, longer if your audience is small.
  • Not to change too many elements of the page

This means you have to get smart about what you are testing. Restraint is important, and trusting yourself and not putting every single design argument up for testing is required.

Field Research: This is most useful when you have some quirky users. Such as people who are searching in difficult environments, people who have a very particular idea about how search works, or people for whom is part of a larger, complex task.

Field research is a simple concept. You go to people’s homes, or work environments, or wherever they use your product, and watch them using it in context of their real, daily tasks.

This information helps you decide what features you need, what expectations you must meet, and perhaps even who’s design you should copy.

However, to do field research effectively, it’s best to rely on a trained researcher. If you do it yourself, you can easy disrupt the natural flow, or ask leading questions, again delivering bad data. If you can’t afford to hire a pro, at least buy Observing the User Experience: A Practitioner’s Guide to User Research, and get some tips on mistakes to avoid.