Situation

In 2018, Amazon acquired a company that provided an engine to help Alexa answer general informational questions such as “Who is the prime minister of the UK?” or “When did the movie Armageddon come out?” As we began to integrate those answers into Alexa’s responses, we noticed that they (and, to be fair, many of the answers from other Alexa components that already existed) did not seem to be the ideal answers for a spoken response. Like modern LLMs, this engine was trained on and designed to provide written responses on the web, which did not always translate perfectly to being spoken. Our leadership felt that these answers were lowering Alexa’s response quality bar and disappointing our customers, even when they were technically correct.

Task

While the knowledge engineers (KEs) responsible for these responses were eager to work with us to improve them, the design team didn’t have any organized guidance to provide. As the lead voice designer for Alexa Information, I was asked to work with my team to assemble guidance that we could provide to the knowledge engineers so that they could structure higher-quality spoken responses.

Action

I led the effort to create the spoken question-answering guidelines for Alexa. I identified that we needed three things:

1. A coherent set of written guidelines that knowledge engineers could use to structure responses,
2. A short (2-hour) training curriculum that we could teach to KEs and others who were creating factual-answer experiences for Alexa, and
3. An article summarizing our guidance that would be published as part of the Alexa Design Guide for other voice/conversation designers.

We began by identifying the core elements of the answer guidelines, which we split into two parts: 1) a general intro to Grice’s Cooperative Principle (aka “Grice’s Maxims”), a set of four guidelines proposed by the philosopher Paul Grice that describe how humans effectively cooperate through communication; and 2) the “anatomy of an answer”, outlining the components of a factual answer response and various rules around them, including implicit confirmation of the question; any necessary clarifications; the answer itself, and any “bonus” relevant information worth including.

Result

Along with one of the other designers, I conducted our training session with knowledge engineers across the Alexa organization, both for the newly-acquired team as well as existing KE teams. We recorded the training, and watching our training session became a standard part of KE onboarding for Alexa.

In addition, I worked with the third designer from our team to publish the summary article as part of the official Alexa Design Guide for internal conversation designers. This article has been referenced frequently throughout its existence, and remains highly relevant today as we shift to LLM technologies.