Proceedings of the Annual Meeting of the Cognitive Science Society

At first glance, humans extract social judgments from faces, in-cluding how trustworthy, attractive, and aggressive they look.These impressions have profound social, economic, and polit-ical consequences, as they subconsciously influence decisionslike voting and criminal sentencing. Therefore, understand-ing human perception of these judgments is important for thesocial sciences. In this work, we present a modifying autoen-coder (ModifAE, pronounced “modify”) that can model andalter these facial impressions. We assemble a face impressiondataset large enough for training a generative model by ap-plying a state-of-the-art (SOTA) impression predictor to facesfrom CelebA. Then, we apply ModifAE to learn generalizablemodifications of these continuous-valued traits in faces (e.g.,make a face look slightly more intelligent or much less aggres-sive). ModifAE can modify face images to create controlledsocial science experimental datasets, and it can reveal datasetbiases by creating direct visualizations of what makes a facesalient in social dimensions. The ModifAE architecture is alsosmaller and faster than SOTA image-to-image translation mod-els, while outperforming SOTA in quantitative evaluations.