Proceedings of the Annual Meeting of the Cognitive Science Society

Computational neuromodeling methods for evaluating representational dynamics involve intricate analysis choices at every stage of the analysis pipeline. Analysis choices for data processing pipelines are generally chosen based upon end to end accuracy metrics and corresponding performance metrics. Psychology research has recently begun to acknowledge the importance of controlling for potential bias introduced by degrees of freedom in data analysis, with specification curve analysis introduced as a principled method for correcting for such biases. In this paper, we conduct a specification curve analysis (SCA) for representational similarity analysis pipelines reported in the literature for fMRI and EEG datasets, respectively. We show that EEG-based RSA analyses are relatively robust to alternative specifications but that fMRI based analyses are not. Using a novel decision-tree analysis to supplement SCA, we present a potentially more robust pipeline for such analyses.