Along with the increasing digitization of society, our personal data has been explicitly or implicitly collected and shared through a plethora of digital devices and online social media services. This personal data has become of vital importance for researchers, designers, and artists to represent an impression of our datafied society and to depict the images of data subjects through various forms of data representations. Meanwhile, the explosive increase of personal data has also been accelerated by commercial or governmental entities behind services or technologies to monetize the customer data or surveil citizens. These entities use the data to categorize and predict our behaviors, preferences, and identities through machines that are designed for services or applications, such as content recommendation and facial recognition.

As our society is increasingly datafied, we see ourselves through our data, which is collected and processed by the machines, for self-representation and self-understanding. Moreover, responses from the machines also affect how we behave and understand ourselves. Within this data-centered human-machine interrelationship, the Human-Data-Machine Loop, the machines see us through available, measurable data obtained from us and through stochastic, algorithmic processes to generalize individuals. Here, uncertainty exists because personal data is not an objective representation of oneself, and the machines are not perfect; they can be erroneous and biased upon the data or humans. These issues of uncertainty are difficult to estimate and represent, and they are problems to be solved, especially in scientific domains. But, this uncertainty perspective can be a creative force or theme in data art. This dissertation proposes an artistic approach to reflecting the uncertainty in the Human-Data-Machine Loop through data art. To this end, this dissertation defines the Human-Data-Machine (HDM) Loop as the main conceptual research framework for viewing our datafied society, along with possible types/sources of uncertainty in the Loop. Second, I propose three types of data art practice based on the HDM framework: Artist-Centered Practice, Artist-Machine Collaborative Practice, and Machine-Centered Generative Practice. Last, this dissertation explains and evaluates the author’s data-driven audiovisual art projects as an empirical case study of each data art practice.

This dissertation aims to contribute to expanding data art practice with the perspective of uncertainty in data practice and to raising the audience's awareness of uncertainty in data practice through artistic approaches.

The rapid development of the current availability of advanced hardware and software is opening up new opportunities for digital creation every day. This situation provides great freedom for new artistic expressions with advanced audio, graphics, interface, and algorithms, including machine learning. However, while our nature is multimodal, these modalities in the digital domain are genuinely separate, and the computational platform allows innumerable varieties of linkages among them. For this reason, the holistic multimodal experience is highly dependent on the design and connection of different modalities. This dissertation explains the properties of coherent digital multimodal instruments and discusses their creative opportunity from the process of music composition and performance. The chapters introduce the related projects with their design process, demonstrating the role of crossmodal correspondence in scientific simulation and proposing a numerical method to evaluate the crossmodal correspondences using the correlation coefficient between the modalities. This dissertation aims to contribute to reorganizing the design process of multimodal instruments beyond the old and recent customs.

With a digitally implemented generative system for artistic purposes, controlling a non-trivial number of parameters is one of the difficulties in the process of temporal composition with the system. Unless done manually for every keyframe in the duration of the work, a kind of interface that facilitates the process is necessary. This research proposes an interface for temporal control on parameters of the generative audio-visual system. The proposed interface operates in `state space' formed by a simplex shape of arbitrary dimension with the barycentric coordinate to define the locations inside. Each vertex of simplex represents a reference point in the parameter space of the system. In this manner, regardless of geometric relation between reference points in parameter space, the artist can locate and use any mixed state of the reference states by simple interpolations. This decoupling of the relative coordinate of the states from the specific values of the parameters makes an abstraction for the artist to perform the manual design of the temporal structure efficiently. The state space is a connection between the generative system and the temporal control. With the proposed interface, three artworks are introduced as proof of concepts. Reconstruction is an installation piece, experimenting with virtual geographical shapes. The Reconstruction is a proof of concept that utilizes the state space model for its automatic control, hinting the possibility of a state space model for connection between the generative system and manual composition. Balanced Movement is a fixed length composition, using dynamic equilibrium from the stochastic system as the primary material of the visual. Balanced Movement utilizes state space to define its manual temporal composition. Wavefront is an immersive audio-visual performance using a mathematical abstraction of an ocean wave as visual material. In Wavefront state space is used to spawn entities in performance time dynamically.

In the practice of media arts, which resides at the intersection of art, engineering and science, seamless communication between disciplines is crucial to creating work that can make an impact in all three areas. The need for a language that enables effective communication between artists, scientists and engineers becomes more prominent as experts from each field strive to push boundaries in their research. In this master’s project, I propose several approaches utilizing abstract representations afforded by interactive visualizations to translate between discipline specific languages, creating works that are understood and valued by researchers in fields ranging from arts and humanities to engineering and data science. I will introduce two case studies in support of each of the proposed approaches, demonstrating their translating effect in both interactive art and data visualization.

One of the key functionalities of an immersive virtual environment (IVE), is to induce presence, or the sense of “being there”, from the user. This is dependent on how the virtual reality (VR) system can manipulate our perception of spatial layout, which involves neurobiological processes of constructing a neural map of the peripersonal space, or the nearby space within our reach. This dissertation aims to understand what presence is, how our brain perceives space, and the importance of peripersonal space in generating presence. Using various iterations of projection-based VR system setups as examples of how to utilize this knowledge, this dissertation will provide insight on how to effectively set up projection-based VR systems with limited resources.

In 1962 Karlheinz Stockhausen’s “Concept of Unity in Electronic Music” introduced a connection between the parameters of intensity, duration, pitch, and timbre using an accelerating pulse train. In 1973 John Chowning discovered that complex audio spectra could be synthesized by increasing vibrato rates past 20Hz. In both cases the notion of acceleration to produce timbre was critical to discovery. Although both composers also utilized sound spatialization in their works, spatial parameters were not unified with their synthesis techniques. This dissertation examines software studies and multimedia works involving the use of spatial and visual data to produce complex sound spectra. The culmination of these experiments, Spatial Modulation Synthesis, is introduced as a novel, mathematical control paradigm for audio-visual synthesis, providing unified control of spatialization, timbre, and visual form using high-speed sound trajectories.

The unique, visual sonification and spatialization rendering paradigms of this disser- tation necessitated the development of an original audio-sample-rate graphics rendering implementation, which, unlike typical multimedia frameworks, provides an exchange of audio-visual data without downsampling or interpolation.

Paideia, a Classical ancient Greek notion of education, encompassed both holistic formal instruction and art (particularly poetry) as means to develop cultural values and understanding. This dissertation examines how the new media arts—as a transdisciplinary and computational field—can be leveraged to fulfill a contemporary notion of paideia in both its practice and instruction, demonstrated through respective artistic and pedagogical proofs-of-concept. Specifically, we analyze the ways in which the computational platform can facilitate the creation and readability of causal relationships across information to embody value, meaning, and holistic thought. We examine how we design affordances—formal qualities that constrain an object’s possible uses by an agent—for guiding participant behavior (observed or enacted) and interpreting meaning as part of experiential learning. We will achieve this through artistic installations and objects. We discuss pedagogical strategies with a cross-disciplinary conceptual framework to support holistic thinking through digital design-based learning. We compare these new efforts toward paideia with ancient Greek media and approaches. Toward imparting values, the cybernetic nature of physical computing and other interactive systems offer a means to model or to incentivize (and thus train for) certain kinds of participant actions and behaviors. We present an original body of performance and installation work designed to cultivate the values of social cooperation and attention to nuance. Toward shaping meaning, we propose that complex non-verbal ideas may be communicated by strategic formal design that 1) follows “natural [somatic] mappings” or culturally established associations and/or 2) explicitly defies them, directing attention through “strangeness.” To this end, through a theoretical framework and artistic proofs-of-concept, we explore how computation facilitates abstraction and metaphor via operations of mapping, analysis, and execution. Lastly, to promote trans-paradigmatic thinking, media arts education and practice through fictional worldmaking can provide a means for organizing complex knowledge and systems. We discuss conceptual framework contributions within the THEMAS (STEAM + creative Humanities) pedagogical model (as originally conceptualized by Marcos Novak). In these ways, building upon ancient roots, this research both informs the field of media arts and technology and informs education through design strategies for (artistic) experiential and making-based learning.

Starting from the notion that every material object/structure embodies the information that enables its use, the computational platform facilitates exploring generative information constructs as ever-changing dynamic structures, comparable to the self-organizing systems found in nature.

A study of forms, patterns, and spaces has been undertaken for creating generative media artworks designed for amplifying the reciprocal relationship between an environment that shapes its content and content that shapes its environment.

A main focus of this research is the generation of unique forms of architecture using agent-based behaviors controlled by self-organizing systems. These unique forms of architecture determine, and are, in turn, determined by, the shape of virtual worlds/environments.

Another important focus of this research is the investigation of software platforms that provide the fundamental instruments that produce the content, and that help make and shape the virtual worlds/environments.

In a series of designs, exhibitions, and studies, a New Media Architectural Methodology is presented.

This dissertation is the study of a series of media art installations that deal with embodied technology which allows humans to interact with computers in different contexts. It explores the social effects of ubiquitous technology and the role for embodied media arts as a critique of interactive digital technologies which are replacing physically present forms of communication with our environment and with each other. In these works I examined the affordances of different physical spaces and observed how the social dynamics were affected. To probe these systems, I am using affective computing (computational expression of emotion) and tangible computing (objects with computational power). Intimate spaces allow for the ability to touch and to hold technology. Here the emotive haptic and emotive sonic response of a robot changed the way that people interacted with and viewed the robot and each other. Whether someone was directly interacting with the robot or simply observing drastically changed the emotion they thought the robot was expressing. Social spaces allow for the ability to interact interpersonally with technology. I explore how interactions change when they are embodied and public, while examining gender roles, female agency, and the line between human and machine. Interactive environments allow for movement and for multiple people interacting with large amounts of information and therefore each other. Personal mobile technology, multimodal interfaces, and distributed interfaces were used to allow multiple people to interact with the system and each other. Through these works, I show that media artworks, which embody technology and give it emotive qualities, critique the way that technology is used and change the way that people interact with it.

Embodied worldmaking explores the transdisciplinary creative intersection of embodied movement practice, known in the field of somatics, with virtual worldmaking performance and media arts practice. This research integrates diverse methods into a framework for creating immersive media performative artworks inspired from the body and the natural world that aim to create spaces of refuge for reconnecting to the whole body and sharing embodied stories of interconnectivity with land. This framework takes the form of the embodied worldmaking process, a set of integration methods prioritizing the body in relationship with creative expression, media arts practice, and interactive technology. Specific methods are shared for connecting embodied movement practice, embodied interaction design, performative virtual worldmaking practice, and creative coding system design.

This practice-based arts research is conducted through the iterated processes of creating poetic virtual worlds and instruments inspired by living ecologies, designing interaction mappings between whole-body sensing systems and virtual ecosystems, and exploring improvisational performance and embodiment practices within these systems. This performance research practice provides the grounding motivation for designing the creative coding system, seer, and the four integrative media artworks of this dissertation. 

The audiovisual creative coding system, seer, was created to empower the process of embodied worldmaking. Autobiographical design methods were used to grow the expressivity of the system to support the media artworks of this dissertation. The system enables expressive and rapid production of instruments and virtual ecologies intended for use in immersive installations and improvisational performance environments. Seer provides useful tools for designing audiovisual worlds for performance, as well as new methods for the iterative design of full-body interaction mappings and composition.