Proceedings of the Annual Meeting of the Cognitive Science Society

Understanding the emergence of social cognition requires continuous, high-resolution data from naturalistic environments, yet human studies are often limited by privacy and longitudinal feasibility. The common marmoset offers a compelling model due to its cooperative breeding and compressed developmental timeline that enables the study of family-level social dynamics. However, existing automated tracking tools struggle with the complex backgrounds, frequent occlusions, and rapid morphological changes inherent in home-cage settings. Here we introduce HOLMES, a deep learning-based system designed for longitudinal 3D tracking of multiple marmosets in naturalistic home-cage environments. By integrating Transformer-based temporal tracking with semantic segmentation and pose estimation, HOLMES achieves robust skeletal reconstruction and stable identity tracking across developmental stages. Applying to longitudinal family recordings, this system supports high-throughput quantification of the fine-grained temporal structure of parent-infant interactions and evolving social coordination, providing a way to link moment-to-moment social behaviors with long-term developmental outcomes.