UC Berkeley

The thesis investigates applications and extensions of reinforcement learning (RL) algorithms to modeling human cognition, and focuses on development of new tools for fitting cognitive models to behavioral data. The first part of the thesis examines the effect of choice abstraction on recruitment of RL mechanisms. This work challenges the basic RL assumption that action space is always finite and defined, and tests the variability in processes that best describe the data when the appropriate choice features are ambiguous (e.g. abstract). Results indicate that when choices of multiple levels of abstraction are plausible, less abstract choices (e.g. simple motor actions) interfere with more abstract choices (e.g. goal selection). Further cognitive modeling and experimental tests showed that working memory (WM) contribution to more abstract choice process was reduced relative to that of RL, potentially due to the use of WM resources for defining the appropriate choice features in the abstract condition. Second project explored the effect of subgoals, the intermediate learning milestones, on learning in the context of hierarchical reinforcement learning (HRL) framework. In this project we operationalized subgoals in a way that removes the features commonly associated with subgoals (novelty, reward associations, frequency) and sought to test whether subgoals contribute to learning hierarchically organized policies and generalization through a pseudoreinforcing effect independent of these features. The results revealed that participants solved the hierarchical task, with data patterns implying the effect of subgoals on behavior; generalization tests showed that generalization of subgoals, under the constraint of our subgoal definition, was possible but predicated on explicit recognition of subgoal features. The third project focused on development of new cognitive model-fitting tool leveraging artificial neural networks (ANN). The results demonstrating ANN efficacy in fitting parameters and identifying models with tractable and intractable likelihoods, with comparable (or better) performance relative to standard methods where standard methods were applicable.