Skip to main content
eScholarship
Open Access Publications from the University of California

UC Santa Barbara

UC Santa Barbara Electronic Theses and Dissertations bannerUC Santa Barbara

Coherent Digital Multimodal Instrument Design and the Evaluation of Crossmodal Correspondence

Abstract

The rapid development of the current availability of advanced hardware and software is opening up new opportunities for digital creation every day. This situation provides great freedom for new artistic expressions with advanced audio, graphics, interface, and algorithms, including machine learning. However, while our nature is multimodal, these modalities in the digital domain are genuinely separate, and the computational platform allows innumerable varieties of linkages among them. For this reason, the holistic multimodal experience is highly dependent on the design and connection of different modalities. This dissertation explains the properties of coherent digital multimodal instruments and discusses their creative opportunity from the process of music composition and performance. The chapters introduce the related projects with their design process, demonstrating the role of crossmodal correspondence in scientific simulation and proposing a numerical method to evaluate the crossmodal correspondences using the correlation coefficient between the modalities. This dissertation aims to contribute to reorganizing the design process of multimodal instruments beyond the old and recent customs.

Main Content
For improved accessibility of PDF content, download the file to your device.
Current View