Proceedings of the Annual Meeting of the Cognitive Science Society

An essential part of cognitive science is designing experimentsthat distinguish competing models. This requires patience andingenuity—there is often a large space of possible experimentsone could run but only a small subset that might yield informa-tive results. We need not comb this space by hand: If we useformal models and explicitly declare the space of experiments,we can automate the search for good experiments, looking forthose with high expected information gain. Here, we presentan automated system for experiment design called webppl-oed.In our system, users simply declare their models and experi-ment space; in return, they receive a list of experiments rankedby their expected information gain. We demonstrate our sys-tem in two case studies, where we use it to design experimentsin studies of sequence prediction and categorization. We findstrong empirical validation that our automatically designed ex-periments were indeed optimal.