We present the Perceptimatic English Benchmark, an open ex-perimental benchmark for evaluating quantitative models ofspeech perception in English. The benchmark consists of ABXstimuli along with the responses of 91 American English-speaking listeners. The stimuli test discrimination of a largenumber of English and French phonemic contrasts. They areextracted directly from corpora of read speech, making themappropriate for evaluating statistical acoustic models (such asthose used in automatic speech recognition) trained on typicalspeech data sets. We show that phone discrimination is corre-lated with several types of models, and give recommendationsfor researchers seeking easily calculated norms of acoustic dis-tance on experimental stimuli. We show that DeepSpeech,a standard English speech recognizer, is more specialized onEnglish phoneme discrimination than English listeners, and ispoorly correlated with their behaviour, even though it yields alow error on the decision task given to humans.

We compare the performance of humans (English and Frenchlisteners) versus an unsupervised speech model in a perceptionexperiment (ABX discrimination task). Although the ABXtask has been used for acoustic model evaluation in previousresearch, the results have not, until now, been compared di-rectly with human behaviour in an experiment. We show that astandard, well-performing model (DPGMM) has better accu-racy at predicting human responses than the acoustic baseline.The model also shows a native language effect, better resem-bling native listeners of the language on which it was trained.However, the native language effect shown by the models isdifferent than the one shown by the human listeners, and, no-tably, the models do not show the same overall patterns ofvowel confusions.