Unhappily Ever After
Once upon a time, there was a kingdom with the noble goal of providing its citizens (in modern parlance, “customers”) with good service while cutting the associated costs. The prince in charge of the project was perplexed: Good service meant being available to help citizens whenever they needed it, yet the royal treasury was limited. How could he complete his mission?
A handsome merchant rode into town selling fantastic devices that promised a solution to this quandary: self-service systems that were always available, giving citizens access to the information and transactions they wanted, all for a fraction of the cost of employing enough people to do the same job. The prince hesitated when he heard the sum the merchant required, but in the end the prince convinced the royal advisers that this would be an investment in serving the citizens on whom the kingdom depended.
The merchant delivered the magical self-service system, as promised, and rode out of town with his bags of gold. The prince presented the system to the royal advisers and, despite his doubts, reported the claims of the merchant about how the system would meet the needs of the citizens.
Next, the prince petitioned the king for permission to consult a wandering wise woman, knowledgeable in the ways of self-service, to look upon the system and pronounce it fit to serve the citizens. The wise woman had asked for only a few gold pieces—much less than the merchant charged, the prince argued—and the wise woman’s blessing would bring them good fortune in all the affairs of the kingdom. The king agreed.
The wise woman came and gazed upon the self-service system and spoke words of advice. But she confounded the prince by withholding her blessing, instead delivering this cryptic message: “The only blessing of value had to come from the citizens themselves.” The prince was left suspecting that the wise woman was just a weird, old hippie living alone in a cave, not a prophet to be consulted during uncertain times. Sensing the prince’s dismay, the wise woman decided to write this column.
OK, back to real life. I apologize for the silly story and for seeming vain for portraying myself as the wise woman. For months I have been trying to find an effective way to write about the limitations of one of my most popular consulting services, and I finally settled on this fairy tale.
The technique in question goes by many names: expert review, heuristic evaluation, user experience assessment. Regardless, this is a method for evaluating a user interface without direct input from users. Instead, the method relies on the judgment and experience of an expert. To conduct a review, the expert assumes the role of a typical user and engages with the system to complete common tasks with the goal of understanding users’ real-world experiences. Those interactions allow the expert to identify positive aspects of the interface as well as areas for improvement.
Expert review is popular with clients, especially in tough economic times, because it is a quick and relatively inexpensive way to get a second opinion about a self-service system. Expert review is a great first step in improving user experience, but it’s just that—a first step. Too many clients ignore its obvious limitations and treat it as a cure-all. They believe that no matter what they’ve done wrong in the project, everything will be fine as long as they get the blessing of an expert.
Let me set the record straight: An expert review is nothing more than the judgments and opinions of one person with vast experience and talent for analysis. Expert review is not quantitative, it doesn’t empirically measure anything, and it is not truly predictive. By definition, expert review cannot tell you how customers will react to your system, whether they will be able to use it to complete their tasks, or whether they’ll like it.
If the expert you’ve hired is good, then the review will deliver a set of meaningful insights and recommendations for improving your system. Still, it’s no substitute for testing with actual users. Any expert worth hiring will tell you what the wise woman told the prince: “It’s not my blessing that you need—it’s that of the customers who use the system.”
So is it worth doing an expert review? Absolutely, as long as you understand that the findings tell only part of the story and that meticulously collected user feedback is the best way to improve user experience. Now I’m going back to my cave.
Susan Hura, Ph.D., is principal and founder of SpeechUsability, a VUI design consulting firm. She can be reached at susan@speechusability.com.