UC San Diego

Despite the ubiquity of computers leading to a steady increase in global music consumption, computing has yet to fundamentally transform the capacity for non-musicians to engage with music. The experience of non-musicians—that of passively listening to music—is largely the same as it was decades prior. In this dissertation, I describe my research which allows non-musicians and musicians alike to create, respond, modify, or otherwise interact with music in new ways. Examples include a system which allows non-musicians to improvise on musical instruments without experience, and a system which automatically choreographs music to provide an interactive listening and dancing experience.

Common to all of my work is the use of predictive models which allow us to efficiently sift through musical spaces to identify promising content. The identified content can then be further curated by humans as part of an interactive musical workflow. To build such predictive models, I use machine learning which seeks to extract and generalize patterns in human-composed music. Specifically, I focus on a subfield of machine learning called deep learning, which is capable of extracting such patterns in high-dimensional music representations including both symbolic and acoustic formats. My research focuses on both advancing the state of the art in deep learning for music (and other types of multimedia), and designing interfaces which allow humans to intuitively benefit from what deep learning systems have discovered about music.