Department of Computational Media

Simulationist interactive narrative systems allow game makers to craft reactive stories driven by simulated characters and their social dynamics. These systems produce narrative experiences that feel more emergent but may lack a coherent plot structure. We explored how to combine the emergent possibilities of social simulation with a procedural narrative system that affords writers strong authorial control over the plot. We did this by developing a Unity extension called Anansi that helps people create social simulation-driven visual novels. It enables users to inject simulation data into their story dialogue using logical queries and parameterized storylets written using Ink. The paper describes an overview of our extension and how we empower writers to drive narrative progression using cascading social effects from player choices.

Automatic visual recognition for photo-based food diaries is increasingly prevalent. However, existing tools in food recognition often focus on food classification and calorie counting, which may not be sufficient to support the variety of food and healthy eating goals people have. To understand how to better design computer-vision-based food diaries to support healthy eating, we began to examine how nutrition experts, such as dietitians, use the visual features of food photos to evaluate diet quality. We conducted an observation and interview study with 18 dietitians, during which we asked the dietitians to review a seven-day photo-based food diary and fill out an evaluation form about their observations, recommendations, and questions. We then conducted follow-up interviews to understand their strategies, needs, and challenges of photo diary review. Our findings show that dietitians used the photo features to understand long-term eating patterns, diet variety, eating contexts, and food portions. Dietitians also adopted various strategies to achieve these understandings, such as grouping photos to find patterns, using color to estimate food variety, and identifying background objects to infer eating contexts. These findings suggest design opportunities for future compute-vision-based food diaries to account for dietary patterns over time, incorporate contextual information in dietary analysis, and support collaborations between nutrition experts, clients, and computer vision systems in dietary review and provide individualized recommendations.

Our demo showcases a choice-based interactive narrative game created to teach responsible conduct of research and research ethics. It re-imagines the experience of a game previously published in the literature, using a content-selection system that dynamically constructs dialogue choices during play. Our goal is to provide players with more opportunities to experience agency than they would have with the original game's hand-authored branching narrative structure. Primarily, our system implements a conversation thread-switching mechanic that allows players to fluidly enter/exit conversation topics as one would in a real-life conversation.

Visual novels are a popular game genre for educational games. However, they often feature pre-authored plot structures that cannot dynamically adjust to the player's progression through learning objectives. Employing procedural storytelling techniques boosts plot dynamism, but this comes at the cost of needing a larger repository of content (dialogue and images) to support different learning progressions and objectives. In this paper, we present postmortem-style case studies describing the lessons we learned from attempting to integrate large-language models (LLMs) and text-to-image models into the development of an educational visual novel about responsible conduct of research. Specifically, we discuss our experiences employing generative AI in our dialogue, character sprite, and background image creation processes.

While the game design and game studies communities have analyzed combat both in specific games and game genres, and while combat is clearly central to many types of games, there is no general account of combat that is portable across diverse games. We provide such an account in the form of criteria which are satisfied by games that players inter- pret as “having combat.” These requirements are eventually fulfilled via operational logics, which tie the game’s observ- able behavior (including its instantial assets) to play expe- riences and cultural knowledge, creating what we refer to as a “combat model.” In addition to establishing a com- prehensive model of combat, making it possible to discuss combat across game genres, this work is the first to describe how complex playable models are constructed from composi- tions of operational logics working in concert; we also define two families of logics which are novel in the literature. This broad model of combat has already proved useful in practice, yielding insights in the analysis of the art game Unmanned; it also promises exciting computational applications in areas such as game design support tools and general game playing.

This experience report designs and introduces an application, PlantNet, which is based on hybrid spaces and combines augmented reality (AR) to help enhance the inclusiveness and accessibility of mobile application design. PlantNet is a hiking-focused application incorporating virtual plant cultivation and collection aimed at users with and without disabilities. This report first introduces the theoretical basis of this application and then introduces its main functions and gameplay. Finally, this report discusses the possibilities of hybrid spaces and AR to enhance the inclusiveness and experience of applications. The report concludes by describing the limitations of the current design and making recommendations to provide a foundation for future design.

BACKGROUND: As entrustable professional activities (EPAs) are implemented in graduate medical education, there is a great need for tools to efficiently and objectively evaluate clinical competence. Readiness for entrustment in surgery requires not only assessment of technical ability, but also the critical skill of clinical decision-making. OBJECTIVE: We report the development of ENTRUST, a serious game-based, virtual patient case creation and simulation platform to assess trainees decision-making competence. A case scenario and corresponding scoring algorithm for the Inguinal Hernia EPA was iteratively developed and aligned with the description and essential functions outlined by the American Board of Surgery. In this study we report preliminary feasibility data and validity evidence. METHODS: In January 2021, the case scenario was deployed and piloted on ENTRUST with 19 participants of varying surgical expertise levels to demonstrate proof of concept and initial validity evidence. Total score, preoperative sub-score, and intraoperative sub-score were analyzed by training level and years of medical experience using Spearman rank correlations. Participants completed a Likert scale user acceptance survey (1=strongly agree to 7=strongly disagree). RESULTS: Median total score and intraoperative mode sub-score were higher with each progressive level of training (rho=0.79, P<.001 and rho=0.69, P=.001, respectively). There were significant correlations between performance and years of medical experience for total score (rho=0.82, P<.001) and intraoperative sub-scores (rho=0.70, P<.001). Participants reported high levels of platform engagement (mean 2.06) and ease of use (mean 1.88). CONCLUSIONS: Our study demonstrates feasibility and early validity evidence for ENTRUST as an assessment platform for clinical decision-making.