Proceedings of the Annual Meeting of the Cognitive Science Society

We propose a method for measuring the consistency of ordinal annotations based on a pre-trained embedding vector space. Intuitively, our method finds a direction in the embedding space along which data points align as closely as possible to their annotated ranks. The proposed approach guarantees a globally optimal solution that is free from approximation errors. Thus, it yields a unique consistency measure given a dataset with human-provided ordinal annotations and a pre-trained embedding model. This feature facilitates a wide range of applications, including not only ordinal prediction but also the unsupervised detection of annotation errors within datasets, as well as consistency assessment of stage-based scales (e.g., whether the transitions "beginner to intermediate" and "intermediate to advanced" form linear progressions in the embedding space) during dataset construction. We evaluate our method using real-world datasets with ordinal annotations to demonstrate its effectiveness.