Lexical retrieval requires selecting and retrieving the most appropriate word from the lexicon to express a desired concept. Few studies have probed lexical retrieval with tasks other than picture naming, and when non-picture naming lexical retrieval tasks have been applied, both convergent and divergent results emerged. The presence of a single construct for auditory and visual processes of lexical retrieval would influence cognitive rehabilitation strategies for patients with aphasia. In this study, we perform support vector regression lesion-symptom mapping using a brain tumor model to test the hypothesis that brain regions specifically involved in lexical retrieval from visual and auditory stimuli represent overlapping neural systems. We find that principal components analysis of language tasks revealed multicollinearity between picture naming, auditory naming, and a validated measure of word finding, implying the existence of redundant cognitive constructs. Nonparametric, multivariate lesion-symptom mapping across participants was used to model accuracies on each of the four language tasks. Lesions within overlapping clusters of 8,333 voxels and 21,512 voxels in the left lateral prefrontal cortex (PFC) were predictive of impaired picture naming and auditory naming, respectively. These data indicate a convergence of heteromodal lexical retrieval within the PFC.