Model-free (MF) and model-based (MB) reinforcement learn-ing (RL) have provided a successful framework for under-standing both human behavior and neural data. These two sys-tems are usually thought to compete for control of behavior.However, it has also been proposed that they can be integratedin a cooperative manner. For example, the Dyna algorithm usesMB replay of past experience to train the MF system, and hasinspired research examining whether human learners do some-thing similar. Here we introduce an approach that links MFand MB learning in a new way: via the reward function. Givena model of the learning environment, dynamic programmingis used to iteratively approximate state values that monotoni-cally converge to the state values under the optimal decisionpolicy. Pseudorewards are calculated from these values andused to shape the reward function of a MF learner in a waythat is guaranteed not to change the optimal policy. We showthat this method offers computational advantages over Dyna intwo classic problems. It also offers a new way to think aboutintegrating MF and MB RL: that our knowledge of the worlddoesn’t just provide a source of simulated experience for train-ing our instincts, but that it shapes the rewards that those in-stincts latch onto. We discuss psychological phenomena thatthis theory could apply to, including moral emotions.

How do we learn to think better, and what can we do to pro-mote such metacognitive learning? Here, we propose that cog-nitive growth proceeds through metacognitive reinforcementlearning. We apply this theory to model how people learn howfar to plan ahead and test its predictions about the speed ofmetacognitive learning in two experiments. In the first experi-ment, we find that our model can discern a reward structure thatpromotes metacognitive reinforcement learning from one thathinders it. In the second experiment, we show that our modelcan be used to design a feedback mechanism that enhancesmetacognitive reinforcement learning in an environment thathinders learning. Our results suggest that modeling metacog-nitive learning is a promising step towards promoting cognitivegrowth.

What is the optimal way to make a decision given that yourtime is limited and your cognitive resources are bounded? Toanswer this question, we formalized the bounded optimal de-cision process as the solution to a meta-level Markov deci-sion process whose actions are costly computations. We ap-proximated the optimal solution and evaluated its predictionsagainst human choice behavior in the Mouselab paradigm,which is widely used to study decision strategies. Our compu-tational method rediscovered well-known heuristic strategiesand the conditions under which they are used, as well as novelheuristics. A Mouselab experiment confirmed our model’smain predictions. These findings are a proof-of-concept thatoptimal cognitive strategies can be automatically derived as therational use of finite time and bounded cognitive resources.

People’s cognitive strategies are jointly shaped by function and computational constraints. Resource-rational analysis lever- ages these constraints to derive rational models of people’s cognitive strategies from the assumption that people make rational use of limited cognitive resources. We present a resource-rational analysis of planning and evaluate its predic- tions in a newly developed process tracing paradigm. In Ex- periment 1, we find that a resource-rational planning strategy predicts the process by which people plan more accurately than previous models of planning. Furthermore, in Experiment 2, we find that it also captures how people’s planning strategies adapt to the structure of the environment. In addition, our ap- proach allows us to quantify for the first time how close peo- ple’s planning strategies are to being resource-rational and to characterize in which ways they conform to and deviate from optimal planning.

Background

Primary care provides most of the evidence-based chronic disease prevention and screening services offered by the healthcare system. However, there remains a gap between recommended preventive services and actual practice. This trial (the BETTER Trial) aimed to improve preventive care of heart disease, diabetes, colorectal, breast and cervical cancers, and relevant lifestyle factors through a practice facilitation intervention set in primary care.

Methods

Pragmatic two-way factorial cluster RCT with Primary Care Physicians' practices as the unit of allocation and individual patients as the unit of analysis. The setting was urban Primary Care Team practices in two Canadian provinces. Eight Primary Care Team practices were randomly assigned to receive the practice-level intervention or wait-list control; 4 physicians in each team (32 physicians) were randomly assigned to receive the patient-level intervention or wait-list control. Patients randomly selected from physicians' rosters were stratified into two groups: 1) general and 2) moderate mental illness. The interventions involved a multifaceted, evidence-based, tailored practice-level intervention with a Practice Facilitator, and a patient-level intervention involving a one-hour visit with a Prevention Practitioner where patients received a tailored 'prevention prescription'. The primary outcome was a composite Summary Quality Index of 28 evidence-based chronic disease prevention and screening actions with pre-defined targets, expressed as the ratio of eligible actions at baseline that were met at follow-up. A cost-effectiveness analysis was conducted.

Results

789 of 1,260 (63%) eligible patients participated. On average, patients were eligible for 8.96 (SD 3.2) actions at baseline. In the adjusted analysis, control patients met 23.1% (95% CI: 19.2% to 27.1%) of target actions, compared to 28.5% (95% CI: 20.9% to 36.0%) receiving the practice-level intervention, 55.6% (95% CI: 49.0% to 62.1%) receiving the patient-level intervention, and 58.9% (95% CI: 54.7% to 63.1%) receiving both practice- and patient-level interventions (patient-level intervention versus control, P < 0.001). The benefit of the patient-level intervention was seen in both strata. The extra cost of the intervention was $26.43CAN (95% CI: $16 to $44) per additional action met.

Conclusions

A Prevention Practitioner can improve the implementation of clinically important prevention and screening for chronic diseases in a cost-effective manner.