Perceived dominance and trustworthiness have both been found to be positive predictors for a candidate's electoral and employment success. On the other hand, compared to male faces, female faces exhibit a much stronger anti-correlation between perceived trustworthiness and dominance. Together, these two phenomena place women at a distinctive disadvantage to men in electoral and work settings. In this study, we conduct computational analyses on a gender- and race-balanced, publicly available face dataset to examine the provenance of the anti-correlation between perceived dominance and trustworthiness in female faces. By identifying and quantifying the facial features that contribute to each social trait, we find that the female anti-correlation stems predominantly from components unique to female faces (83\%), with the lip region being the main contributor (23\%). Visualization of face featural modifications show that the corners of the mouth curve up and down in opposite directions for perceived trustworthiness and dominance, respectively, in female faces, but in orthogonal directions the same in male faces. By correlating gender specific models with perceived demographic information, we find that female dominance ($F_D$) and trustworthiness ($F_T$) are correlated in opposite directions along most perceived gender, age and race-related demographic dimensions. Male dominance ($M_D$) and trustworthiness ($M_T$) , on the other hand, are correlated in the same direction along race-related dimensions, but otherwise share no significant demographic dimensions (age and gender). In particular, perceived sexual dimorphism strongly drives $F_D$, $F_T$, and $M_D$, but is absent for $M_T$, indicating sexual dimorphism is a strong contributor to the female anti-correlation.