We are highly tuned to each other’s visual attention. Perceiving the eye or hand movements of another person can influence the timing of a saccade or a reach of our own. However, it is not clear whether the effect of social cues is due to the appearance of the cue – a hand or an eye - or the belief that the cues are connected to another person. In two experiments we investigated this question using a spatial cueing paradigm and measuring the inhibition of return of visual attention. When participants believed that a cue stimulus – a red dot – reflected the attentional focus of another person via an eye tracker, they responded differently to when they believed its location was determined by a computer. Despite previous claims that they are ‘blind’ to such factors, when a cue was imbued with a social context it exerted a stronger influence over low-level visual attention.