Proceedings of the Annual Meeting of the Cognitive Science Society

The goal of the research reported in this paper is to uncover the cognitive processes involved in designing complex experiments in contemporary biology. Models of scientific reasoning often assume that the experimental design process is primarily theoretically constrained. However, designing an experiment is a very complex process in which many steps and decisions must be made even when the theory is fully specified. We uncover a number of crucial cognitive steps in experimental design by analyzing the design of an experiment at a meeting of an immunology laboratory. Based on our analysis, we argue that experimental design involves the following processes: unpacking and specifying slots in possible experimental designs, locally evaluating specific components of proposed designs, and coordinating and globally evaluating possible experimental designs. Four sets of criteria guide local and global evaluation: ensuring a robust internal structure to the experiment, optimizing the likelihood experiments will work, performing costs/benefits analyses on possible design components, and ensuring acceptance of results by the scientific community. Our analyses demonstrate that experimental design is constrained by many non-theoretical factors. In particular, the constant threat of error in experimental results lies behind many of the strategies scientists use.