Lessons Learned from Deploying Natural Language Call Routing
What is natural language really? What are the conditions for a positive business case? Which principles should be considered in the design of such systems? What else can be learned from recent deployments?
Driven by major new deployments, natural language has received renewed attention. Building on contributions by Deborah Dahl(1) and Walter Rolandi(2) in the May/June issue earlier this year, the lessons to be learned stem from various deployments of natural language call routing.
Natural Language Revisited
Since everybody means something different when they talk about natural language, let me begin by clarifying the various uses of the term in the speech technology field. Natural language is often used in one of the three contexts listed in Table 1 below.
Table 1 : Various uses of the term natural language
Applications employ open-ended prompts to route the call to specialized agent pools and self-service applications. Routing, understood as getting callers to the right specialized agent or self-service application, is one of the key problem areas for many call center IVRs. Simply converting touchtone into directed speech menus does not really solve the problem. Even if callers are willing to co-operate, they sometimes are unable to match their reason for calling with any of the options presented in a menu. The evolution from a touchtone menu, via a directed speech menu, to natural language call routing is easier to see in an example. Consider the examples in Table 2 for customers who need to disconnect service because they cannot afford the service any more. Clearly, with either menu, some customers will pick moving, while others will pick other.
Natural language call routing addresses this problem by allowing callers to describe the reason for their call in their own words. The open-ended prompt effectively shifts the burden of understanding from the caller to the natural language technology working behind the scenes. Research has shown that, in addition to reducing routing errors due to picking the wrong option, natural language call routing (3,4):
- Flattens menu hierarchies to one single prompt, reducing the time spent in the IVR
- Increases willingness of callers to co-operate and respond
- Produces greater caller satisfaction
Table 2: Evolution from touchtone to speech menus to open-ended prompts
Is Natural Language right for my call center?
While Dahls article suggested that natural language call routing technology works and is ready for prime time, it did not address the business case. Natural language is more expensive than directed dialog speech recognition because of higher license fees and professional services. Therefore, the next logical question that you might ask is Is natural language right for my call center?
The key issue for decision makers in call centers is whether it makes business sense. The following are some of the conditions which will make a positive business case:
- Multiple, distinct routing destinations - Distinct in this context means that not getting a caller to the correct destination leads either to a transfer (if the caller was routed to the wrong specialized agent pool) or a lost opportunity to self-serve the call in the IVR.
- Size of the call center - The total yearly call volume should be in the millions of calls.
- Above-average value of correct routing - Even if the above conditions dont apply to your call center, the business case may still be positive if your agent costs are very high, or for other reasons, such as premium value of customer retention.
It doesnt take dozens of destinations for natural language to provide benefits. The Comparative Study of Speech in the Call Center3 compares call routing via menus versus an open-ended prompt proving a benefit with just eight routing destinations. While the above rules of thumb may suggest you should consider natural language call routing, building a solid business case thats based on data from your call center is crucial to minimize risk and maximize your return (ROI). This leads you to the lessons learned.
Lesson #1: Drive the business case from analyses of end-to-end call data
Quantifying the business benefit of IVR routing is difficult because the typical ROI model for deploying speech, which focuses on fully self-served calls, cannot be applied directly. Even without fully self-serving a call, there is benefit to getting callers to the right specialized agent, assuming that your call center employs multiple distinct agent queues. Each misrouted call wastes both agent and caller time. The first agent has to diagnose the problem enough to realize that the call has to be transferred, and then decide where to transfer it. Following the transfer, the caller has to wait on hold and explain the problem a second time. Typically, anywhere between one to three minutes are wasted per misrouted call. That quickly adds up to a large savings potential. But routing is equally important to get callers to use self-service applications. Many self-service applications do not reach their potential because callers are getting lost in menus or frustrated before reaching them.
But how do you build a solid business case for natural language call routing? When our parent company deployed BBN Technologys natural language call routing technology seven years ago, they did not buy into the standard business case based on estimates of increasing self-service, rather they based it on BBNs methodology for evaluating and optimizing IVR business benefit based on end-to-end calls5. The key ideas of this methodology are:
- Measure routing by following calls from the IVR interaction through to the agent-caller dialog, to determine whether or not the IVR routed the caller to the correct place.
- Quantify IVR benefit as average agent seconds saved per call, instead of as number of calls deflected from call center agents.
- Credit the IVR for partial automation benefit.
Partial automation saves time by offloading routine tasks, such as capturing account numbers, routing, and delivery of information, regardless whether the call is or is not ultimately handled by an agent. Quantifying routing benefit based on end-to-end calls is necessary because only the agent-caller dialog provides the evidence of whether a call was or was not routed correctly. Misrouting rates are often measured based on statistics collected from each agent pool. These statistics are not based on total inbound traffic and missed pre-agent routing problems that still can have an effect on a substantial percentage of all callers.
An analysis of end-to-end calls yields a second key to the puzzle of building the business case and maximizing ROI of speech: the distribution of why customers are calling, or simply call-reason distribution. In measuring call-reason frequencies it is important to do so relative to all calls handled by the IVR, not just calls handled by agents. The call-reason distribution is valuable because it:
- Provides estimates of the upper bounds on IVR benefit
- Allows you to focus on the high-value areas
- Enables designers to consider the true frequencies throughout the design
Lesson #2: Focus on the high-value call reasons
Many call centers face new challenges as business requirements force them to make more and more distinctions in handling a call. For example, additional agent pools are introduced to handle specific caller populations or new products, or the push to save costs by increasing self-service automation leads to incorporating growing portfolios of self-service applications in the IVR. Designers may look to open-ended prompts as a remedy to bushy menu trees. While flattening menu trees is indeed one of the benefits of employing an open-ended prompt, this benefit should not be misinterpreted as a blank cheque to arbitrary complexity. Rather, even when employing natural language call routing it is recommended that you never lose sight of the time-tested KISS principle (Keep It Simple, Stupid!). Working hard to control complexity and focusing on the high-value call reasons throughout the application design leads to a simpler system, lower development and maintenance costs, higher utilization of self-service applications, and thus higher ROI.
Figure 1: Call-reason distribution example
In 2001 Our client developed a speech-enabled application to allow callers to order telephone directories. BBNs analyses of end-to-end calls revealed that less than one percent of all calls were about ordering directories, not to mention that only a trickle of 0.1 percent of calls reached this new self-service application due to the rather complex menu tree. Hence, the new application provided little or no benefit. Whats the answer to this problem in the spirit of focusing on high-value and ROI? Employ natural language call routing to get more callers to the directory ordering application? No! High-value issues are typically associated with call reasons that represent a large percentage of all calls, and can be self-served or have to be handled by specialized agents. Figure 1 suggests that account balance inquiries in payments are high frequency and self-serviceable, while ordering directories is not even among the top 10 call reasons. In deploying natural language call routing, the recommendation was to focus on making it easy to get to the balance and payment applications.
Lesson #3: Leverage knowledge of limitations to maximize overall effectiveness
Just as with speech recognition in general, optimal use of natural language call routing technology requires knowledge of its limitations, and skillfully leveraging this knowledge in the design of the application. Here are some of these limitations and how to circumvent them with good design:
First, natural language call routing employs statistical algorithms: a statistical classifier determines the reason of a call based on speech recognition output. Such classifiers work better on the more frequent call reasons, as can be seen in the accuracy-frequency plot shown in Figure 2. Notable exceptions are junk utterances that dont contain any reasonable description of the call reason (highlighted by the red arrow in the figure), and call reasons that in the callers mind are confusable. Low frequency call reasons offer diminishing returns: while basically all top 10 call reasons are classified at least 50 percent correct (to the right of the dotted line in Figure 2), accuracy on low frequency call reasons can be very low. Confusable distinctions that are high value and cannot be eliminated must therefore receive special care in designing the dialog that follows the open-ended prompt.
Figure 2: Accuracy-Frequency distribution of a statistical natural language call router, configured to distinguish among more than 40 different call reasons, and evaluated on live calls.
Second, distinguishing many different categories is inherently limited by how distinctively callers describe their problem in response to an open-ended prompt. Comparatively few responses are very detailed, and even fewer contain more than one piece of information. The distribution of utterances in response to an open-ended prompt looks very similar to the call-reason distribution shown in Figure 1: only a few utterance categories occur more than five percent of the time, and many utterances are unspecific, such as, I have a question about my bill, and, I wanna order service. Hence, you cannot rely on the open-ended prompt to make fine-grained distinctions. If such distinctions are associated with high-value, you may have to employ follow-up prompts to elicit additional information. And keeping it simple means that you dont have to worry about low-value distinctions, as handling the call by the default agent is always sufficient.
Finally, due to ambiguity inherent among certain sets of call reasons, sometimes the callers description of the problem does not accurately reflect the root cause of the problem. One example of such an ambiguous call reason is connectivity in the context of high-speed Internet access. Several root causes can lead to symptoms that are to the caller identical to a connectivity problem, including incorrect installation, incomplete service activation, and overdue accounts. So sometimes you can take what the caller is saying only as a cue to what the problem might be. Luckily, such ambiguous call reasons can be identified from analyses of end-to-end calls.
Natural language call routing is ready for prime time and provides significant benefits. The business case for your call center can be explored with an analysis of routing benefit based on end-to-end calls. Once you have made a decision to move forward, design is crucial to success just as with other speech applications. Compared to directed speech menus, the challenge shifts from designing menu hierarchies to defining the set of call reasons and skillfully embedding the open-ended prompt in a dialog that maximizes effectiveness overall. Analyses of end-to-end calls help designers tackle the key challenges in deploying this advanced technology, including:
building a solid business case
focusing on the high-value call reasons
designing a superior call-flow
But above all, avoid excessive complexity, because it will only confuse your customer and lead to higher deployment costs. Natural language call routing works and, when deployed with care, provides substantial ROI to your company.
Bernhard Suhm is a speech scientist and usability specialist in the Call Center Solutions group at BBN Technologies. He can be reached at firstname.lastname@example.org .
- Dahl, D. (2004, May/June). Is Natural Language Real? Speech Technology Magazine, 9(3), 34.
- Rolandi, W. (2004, May/June) Whats Natural about Natural Language Processing? Speech Technology Magazine, 9(3), 37.
- Bers, J., B. Freeman, D. Getty, K. Godfrey, D. McCarthy, P. Peterson and B. Suhm. A Comparative Study of Speech in the Call Center: Natural Language Call Routing vs. Touch-Tone Menus.CHI 2002 Conference, 1, 283-290.
- Bers, J., D. McCarthy and B. Suhm. (2001, November/December). Please Tell Me Briefly the Reason for your Call Understanding Natural Language Call Routing. Speech Technology Magazine, 6, 32.
- Peterson, P. and B. Suhm. (2002). A Data-driven Methodology for Evaluating and Optimizing Call Center IVRs. International Journal of Speech Technology, 5(1), 23-37.