Proceedings of the Annual Meeting of the Cognitive Science Society

We present the Perceptimatic English Benchmark, an open ex-perimental benchmark for evaluating quantitative models ofspeech perception in English. The benchmark consists of ABXstimuli along with the responses of 91 American English-speaking listeners. The stimuli test discrimination of a largenumber of English and French phonemic contrasts. They areextracted directly from corpora of read speech, making themappropriate for evaluating statistical acoustic models (such asthose used in automatic speech recognition) trained on typicalspeech data sets. We show that phone discrimination is corre-lated with several types of models, and give recommendationsfor researchers seeking easily calculated norms of acoustic dis-tance on experimental stimuli. We show that DeepSpeech,a standard English speech recognizer, is more specialized onEnglish phoneme discrimination than English listeners, and ispoorly correlated with their behaviour, even though it yields alow error on the decision task given to humans.