UC Berkeley

This dissertation examines the capacity of interactive gesture to contribute to discourse structure independent of accompanying speech. The close relationship between gesture and speech in face-to-face interaction is, at this point, well-established and accepted, especially within gesture studies and certain linguistic frameworks (e.g. Embodied Construction Grammar and Embodied Conversation Analysis). However, the integration of gesture into formal linguistic theory more generally is still in its early stages of development. This is especially true for interactive gesture and formal theories of discourse structure. To the author’s knowledge, this dissertation serves as the first in-depth exploration of the ability to formalize a theory of interactive meaning in gesture using a predictive model.

The proposed model is built upon the use and management of an ’Interaction Space’ – the physical space in which co-present interlocutors interact and the metaphoric space in which they co-construct a goal-based, hierarchical discourse structure together. I propose an inventory of management actions performed upon this space, each of which is enacted by a particular kind of interactive gesture. Present, Refer and Remove actions constitute the three primary actions performed by interactive hand gestures and serve to introduce, organize, and remove discourse topics as metaphoric objects from the Interaction Space. Engage and Disengage actions are the two primary actions performed by non-manual gestures and serve to manage interlocutors’ roles as speaker or addressee throughout a discourse. Three additional management actions, Separate, Combine and Request are composite actions used to achieve more complex forms of discourse management, such as expressing a particular discourse relation or the desire for an interlocutor to perform particularactions in response.

After an initial explanation of the model and associated formalism, I apply the model in three case studies. Each case study looks at the use of gesture to express a particular discourse move – topic-shifting, digression, and specification, as co-expressed by the lexical discourse markers, anyway, by the way, and here’s the thing. All data comes from interviews and monologues on the American talk show The Late Show with Stephen Colbert and were collected using UCLA’S Communication Studies Archive in collaboration with the Red Hen Lab. Through quantitative and close qualitative analyses, I demonstrate the capacity of interactive gesture to perform discourse management, and the ability of the Interaction Space to parsimoniously model how this is done.

Because of the interdisciplinary nature of this dissertation, I draw from a range of theories in my analyses and discussions. For describing the form and function of gesture, I rely on action schematic approaches to gesture meaning (Cienki 2013; Mittelberg 2018; M ̈uller 2017). For interpreting gestures in context, I use Conceptual Metaphor Theory (Lakoff & Johnson 1980) and Blending (Fauconnier & Turner 1998). For analyzing discourse structure, I borrow from question-based approaches (van Kuppevelt 1995; Roberts 1996, 2012). It is uncommon to find such disparate frameworks together in a single work. In bringing these approaches together, I hope to demonstrate the value of inter-theory and inter-field dialoguesto the development of a comprehensive model of multimodal language use.