Department of Computational Media

Computerized microscopes improve repeatability, throughput, antisepsis, data analysis and data sharing in the biological laboratory, but these machines are cost-prohibitive in most academic environments. This is a barrier into collecting the large and consistent datasets required for machine learning analyses of microscopy data. We demonstrate hardware modifications and software to bring the features of modern computerized microscopes to decades-old legacy laboratory inverted microscopes. We demonstrate automation of X-Y positioning, focus stacking, image acquisition and image storage.

Simulationist interactive narrative systems allow game makers to craft reactive stories driven by simulated characters and their social dynamics. These systems produce narrative experiences that feel more emergent but may lack a coherent plot structure. We explored how to combine the emergent possibilities of social simulation with a procedural narrative system that affords writers strong authorial control over the plot. We did this by developing a Unity extension called Anansi that helps people create social simulation-driven visual novels. It enables users to inject simulation data into their story dialogue using logical queries and parameterized storylets written using Ink. The paper describes an overview of our extension and how we empower writers to drive narrative progression using cascading social effects from player choices.

Visual novels are a popular game genre for educational games. However, they often feature pre-authored plot structures that cannot dynamically adjust to the player's progression through learning objectives. Employing procedural storytelling techniques boosts plot dynamism, but this comes at the cost of needing a larger repository of content (dialogue and images) to support different learning progressions and objectives. In this paper, we present postmortem-style case studies describing the lessons we learned from attempting to integrate large-language models (LLMs) and text-to-image models into the development of an educational visual novel about responsible conduct of research. Specifically, we discuss our experiences employing generative AI in our dialogue, character sprite, and background image creation processes.

Our demo showcases a choice-based interactive narrative game created to teach responsible conduct of research and research ethics. It re-imagines the experience of a game previously published in the literature, using a content-selection system that dynamically constructs dialogue choices during play. Our goal is to provide players with more opportunities to experience agency than they would have with the original game's hand-authored branching narrative structure. Primarily, our system implements a conversation thread-switching mechanic that allows players to fluidly enter/exit conversation topics as one would in a real-life conversation.

Working in a stem cell laboratory necessitates a thorough understanding of complex cell culture protocols, the operation of sensitive scientific equipment, adherence to safety standards, and general laboratory etiquette. For novice student researchers, acquiring the necessary specialized knowledge before their initial laboratory experience can be a formidable task. Similarly, for experienced laboratory personnel, efficiently and uniformly training new trainees to a rigorous standard presents a significant challenge. In response to these issues, we have developed an educational and interactive virtual cell culture environment. This interactive virtual lab aims to equip students with foundational knowledge in maintaining cortical brain organoids and to instill an understanding of pertinent safety procedures and laboratory etiquette. The gamification of this training process seeks to provide laboratory supervisors in highly specialized fields with an effective tool to integrate students into their work environments more rapidly and safely.

Automatic visual recognition for photo-based food diaries is increasingly prevalent. However, existing tools in food recognition often focus on food classification and calorie counting, which may not be sufficient to support the variety of food and healthy eating goals people have. To understand how to better design computer-vision-based food diaries to support healthy eating, we began to examine how nutrition experts, such as dietitians, use the visual features of food photos to evaluate diet quality. We conducted an observation and interview study with 18 dietitians, during which we asked the dietitians to review a seven-day photo-based food diary and fill out an evaluation form about their observations, recommendations, and questions. We then conducted follow-up interviews to understand their strategies, needs, and challenges of photo diary review. Our findings show that dietitians used the photo features to understand long-term eating patterns, diet variety, eating contexts, and food portions. Dietitians also adopted various strategies to achieve these understandings, such as grouping photos to find patterns, using color to estimate food variety, and identifying background objects to infer eating contexts. These findings suggest design opportunities for future compute-vision-based food diaries to account for dietary patterns over time, incorporate contextual information in dietary analysis, and support collaborations between nutrition experts, clients, and computer vision systems in dietary review and provide individualized recommendations.

While the game design and game studies communities have analyzed combat both in specific games and game genres, and while combat is clearly central to many types of games, there is no general account of combat that is portable across diverse games. We provide such an account in the form of criteria which are satisfied by games that players inter- pret as “having combat.” These requirements are eventually fulfilled via operational logics, which tie the game’s observ- able behavior (including its instantial assets) to play expe- riences and cultural knowledge, creating what we refer to as a “combat model.” In addition to establishing a com- prehensive model of combat, making it possible to discuss combat across game genres, this work is the first to describe how complex playable models are constructed from composi- tions of operational logics working in concert; we also define two families of logics which are novel in the literature. This broad model of combat has already proved useful in practice, yielding insights in the analysis of the art game Unmanned; it also promises exciting computational applications in areas such as game design support tools and general game playing.