UC Santa Cruz

In my dissertation I investigated how we expect artificial agents (e.g., chatbots, voice assistants, robots) to ground with people. I focused on two aspects of grounding: (a) how people interpret errors from an agent and (b) how people expect agents to contribute to resolving errors in communication. Two experiments inform how we expect technology to engage in navigating grounding in a conversation. Additionally, these experiments offer insight into how similarly we want technology to mimic human speech behaviors or if we have differing expectations for how technology should communicate.Experiment 1 uses a Wizard of Oz (WOz) design where a voice assistant (VA), in this case an Alexa, commits phoneme level speech errors (swaps, anticipations, and perseverations) across three between-subjects conditions: (1) no errors (no speech errors committed), (2) errors (speech errors are committed), and (3) errors with correction (speech errors are committed and immediately self-corrected with the phrase “[error] I mean [correction]”). Another factor, context, is examined for its effect on VA error perception. This factor has two within-participants conditions: (1) a task-oriented context consisting of selecting a tangram from an array based on VA descriptions and (2) a social oriented context consisting of exchanging ice breaker questions with the VA. Unknown to the participant, in all conditions the VA’s responses will be controlled by a researcher. I measured participants’ attitudes toward technology, perceptions of the agent, and engagement with the agent. I found that while errors were perceived similarly to people – that is, some go unnoticed – the use of errors did not increase human-like perceptions, positive affect, nor engagement compared to no errors present. However, when errors were corrected participants rated the VA as less intelligent. This study informs us that while errors are expected of people, they are not expected of VAs and bringing attention to errors by correcting them can lead to negative perceptions of the VA.Experiment 2 also uses a WOz design. Here the VA used repairs to attempt to ground a conversation. In this study, participants listened to ten short mini-lectures spoken by the VA and answered a question after each lecture. These lectures were designed to be difficult and encouraged signals of confusion such as a (“Huh?”, “What”, “Um/uh”, or a long pause). In response to these signals the VA attempted to clarify the confusion by using a repair (repeat, rephrase or topic shift). As in the previous study, unknown by the participant all VA responses are controlled by a researcher. I measured participants’ attitudes toward technology, perceptions of the agent, engagement with the agent, as well as the effectiveness and appropriateness of the repairs. Participants showed sufficient signs of confusion in response to the lectures. Overall, repairs were not perceived as helpful. The different repair types also did not differ from each other. The number of repairs produced also did not significantly impact participant’s perceptions of the VA. However, repairs were not harmful either: The attempt to repair was not rated more negatively compared to no repair given. This means that even if a VA repairs poorly and is unsuccessful it is not seen significantly more negatively than one that does not try to help. These results show that while people have high expectations for successful grounding, they are also tolerant of artificial agents that make an attempt, even an ineffective one.Together these studies show that people may be more tolerant of missed mistakes compared to admitted mistakes, but failing at trying to help has no negative consequences compared to not trying at all. Future advancements in communication-based technology should encourage technology to work with people to support its limited grounding capabilities.