We tested aphasic patients’ comprehension of actions to examine processing deficits in the linguistic and non-linguistic domains and their lesion correlates. 29 left-hemisphere injured patients and 18 age-matched control subjects matched pictured actions (with the objects missing) or their linguistic equivalents (printed sentences with the object missing) to one of two visually-presented pictures of objects. Aphasic patients performed poorly not only in the linguistic domain but also in the non-linguistic domain. A subset of the patients, largely consisting of severe and non-fluent aphasics, showed a greater deficit in the linguistic domain compared with the non-linguistic domain and across the patient group, deficits in the linguistic and non-linguistic domains were not tightly correlated. Poor performance in pantomime interpretation was associated with lesions in the inferior frontal, premotor and motor cortex, a portion of somatosensory cortex, and the caudate, while poor reading comprehension of actions was associated with lesions around the anterior superior temporal lobe, the anterior insula and the anterior portion of the inferior parietal lobe. Lesion size did not correlate with deficits. The lesion results for pantomime interpretation deficits demonstrate that lesions in the frontal component of the human analog of the “mirror neuron system” are associated with deficits in non-linguistic action understanding. For reading comprehension deficits, the lesion correlates are brain areas known to be involved in linguistic tasks including sentence processing and speech articulation; the parietal lesion site may also correspond to a subpart of the human mirror neuron system. These results indicate that brain areas important for the production of language and action are also recruited in their comprehension. Similar findings have been reported in electrophysiological and neuroimaging studies. Our findings now also lend neuropsychological support to an embodied view of brain organization for action processing.