Proceedings of the Annual Meeting of the Cognitive Science Society

Vectors derived from patterns of co-occurrence of words inlarge bodies of text have often been used as representations ofsome aspects of the meanings of different words. Generally,the distance between such vectors is used as a measure of thesemantic similarity between the word meanings theyrepresent. One important way of evaluating the performanceof these vectors has been to use them to answer vocabularymultiple choice questions (MCQs) where the participant isasked to judge which of several choice words is closest inmeaning to a stem word. The existing vocabulary MCQ testsused in this way have been very useful but there are somepractical problems in their use as general evaluationmeasures. Here, we discuss why such tests remain usefulevaluation measures, introduce a new vocabulary test,evaluate several current sets of semantic vectors using thenew test and compare their performance to human data.