From only brief exposure to a face, individuals spontaneously categorize another's race. Recent behavioral evidence suggests that visual context may affect such categorizations. We used fMRI to examine the neural basis of contextual influences on the race categorization of faces. Participants categorized the race of faces that varied along a White-Asian morph continuum and were surrounded by American, neutral, or Chinese scene contexts. As expected, the context systematically influenced categorization responses and their efficiency (response times). Neuroimaging results indicated that the retrosplenial cortex (RSC) and orbitofrontal cortex (OFC) exhibited highly sensitive, graded responses to the compatibility of facial and contextual cues. These regions showed linearly increasing responses as a face became more White when in an American context, and linearly increasing responses as a face became more Asian when in a Chinese context. Further, RSC activity partially mediated the effect of this face-context compatibility on the efficiency of categorization responses. Together, the findings suggest a critical role of the RSC and OFC in driving contextual influences on face categorization, and highlight the impact of extraneous cues beyond the face in categorizing other people.