What Associative Learning in Insects Tells Us about the Evolution of Learned and Fixed Behavior

Contemporary models for the evolution of learning suggest that environmental predictability plays a critical role in whether learning is expected to evolve in a particular species, a claim originally made over 50 years ago. However, amongst many behavioral scientists who study insect learning, as well as amongst neuroscientists who study the brain architecture of insects, a very different view is emerging, namely that all animals possessing a nervous system should be able to learn. More specifically, the capacity for associative learning may be an emergent property of nervous systems such that, whenever selection pressures favor the evolution of nervous systems, for whatever reason, the capacity for associative learning follows ipso facto. One way to reconcile these disparate views of learning is to suggest that the assumed default in these evolutionary models, namely the non-learning phenotype, is incorrect: The ability to learn is, in fact, the default but, under certain conditions, selection pressures can override that ability, resulting in hard-wired, or considerably less plastic, responses. Thus, models for the evolution of learning actually may be models for the conditions under which inherent plasticity is overridden. Moreover, what have been revealed as the costs of learning in insects may, instead, be costs associated with cognitive abilities that go beyond forming simple associations – cognitive abilities that researchers are just now beginning to reveal. best knowledge, no one has claimed to demonstrate associative learning (with appropriate associative controls) in any member of that insect order. For example, associative learning has not been demonstrated in bristletails, Archaeognatha, the first order listed above. This table is not intended to be an exhaustive list of all papers reporting associative learning in insects. We provide no more than two references for any single insect species, and no more than a few species within a single family. Nonetheless, the figure is designed to expand the information provided in the text, illustrating both the diversity of associative learning in the Class Insecta, as well as obvious gaps in our knowledge.

If we accept that learning is a "fundamental principle of brain functionality" (Greenspan, 2007, p. 649), what, then, shall we make of models for the evolution of learning, models that, in some cases, predict quite accurately the conditions under which animals are capable of forming learned responses, or under which they rely, instead, on hard-wired responses (e.g., Dunlap & Stephens, 2009)? How might we reconcile what appear to be the disparate views of neuroscientists and evolution of learning theorists? In this paper we offer a brief overview of the ways in which theorists have approached the evolution of learning; we provide a snapshot of the research on associative learning in insects; we explore the neuroscientific conundrum that lies at the heart of evolution of learning models; and, finally, we propose a solution to that conundrum.
Elsewhere we have described this same discrepancy between neuroscientific research and models for the evolution of learning ; the current paper is our attempt to update what is known about associative learning in insects, to provide current information about other cognitive abilities that insects possess, and to refine our proposal about how to reconcile the disparate views of neuroscientists and theorists who propose evolution of learning models.

The Evolution of Learning and the "Predictability" Requirement
Early theorizing about the conditions that would favor the evolution of learning emphasized a changing but relatively predictable environment, conditions that would guarantee that the benefits of predictability would exceed the costs that learning would impose. For example, Johnston (1982) identified six potential costs of learning: (1) delayed reproductive effort, which could translate into lowered reproductive success; (2) increased parental investment in each offspring, which could lower the number of offspring produced; (3) increased juvenile vulnerability, sometimes labeled start-up costs, which could place additional burdens on parents; and, (4) developmental fallibility, which could have adverse effects on both parents and offspring, depending on the severity of the errors or omissions. Two additional costs proposed by Johnston (1982) that are especially relevant to our argument -and to which we return later in this paper -are: (5) greater complexity of the genome; and, (6) greater complexity of the central nervous system. Given these costs, learning would be expected to evolve only when the environment changed sufficiently so that hard-wired responses would eventually become a handicap, but did not change so much that learned cues would eventually become unreliable. At about this same time, the notion of predictability figured prominently in theoretical papers highlighting the adaptive nature of associative learning (e.g., Domjan, 1987;Domjan & Hollis, 1988;Hollis, 1982;Moore, 1973;Staddon, 1983). Stephens (1991Stephens ( , 1993, however, challenged the view that predictability arose from a single global source, preferring instead to differentiate between two very different origins, namely within-generation predictability, that is, whether what is learned today can be used tomorrow, and between-generation predictability, that is, whether the environment is similar across generations (see Figure  1). According to this model, learning is favored only when between-generation predictability is low -that is, the environment of offspring differs dramatically from that of their parents -but within-generation predictability is high and, thus, learned cues remain reliable.
More recently, Dunlap and Stephens (2009) have added mathematical rigor to this model and have tested its predictions in a Drosophilia protocol developed by Mery andKawecki (2002, 2004) in which successive generations of fruitflies are subjected to differing experimenter-manipulated environments. These environments favor either hard-wired or learned responses. The new model predicts that when the reliability of current environmental cues (Variable 1, or V1) is greater than the reliability of a fixed response (Variable 2, or V2), a learning phenotype emerges over successive generations of Drosophilia. Notice that in this mathematical model, V1 and V2 appear to map onto the two sources of predictability in the original matrix, namely withingeneration predictability, or V1, and between-generation variability, or V2. But, again, the only cell in the matrix in which V1>V2 is the learning cell. Dunlap and Stephens (2009) have been able to demonstrate quite convincingly that the capacity for learning is lost when fixed patterns of behavior become more reliable. Indeed, they have been able to generate learning and non-learning phenotypes with remarkable precision. However, according to this model -and, for that matter, all other models for the evolution of learning, whether they emphasize predictability or not (e.g., Bergman & Feldman, 1995;Borenstein, Feldman, & Aoki, 2008;Boyd & Richerson, 1985;Irwin & Price, 1999;Nakahashi, 2010) -the capacity for learning emerges only when certain conditions allow it. In other words, the non-learning phenotype in all of these models is the default condition. Interestingly, the history of associative learning research in insects, to which we turn next, not only parallels the history of associative learning research in vertebrates, but also provides some insights about this view that the capacity to learn emerges only under the appropriate environmental conditions. Stephens' model for the evolution of learning. Predictability arises from two separate sources, namely within-generation predictability, that is, whether what is learned today can be used tomorrow, and between-generation predictability, that is, whether the environment is similar across generations. This model, originally proposed by Stephens (1991Stephens ( , 1993 later was revised by Dunlap & Stephens (2009) who modeled and subsequently tested the effects of these two sources of predictability, labeling them Variable 1 (V1) and Variable 2 (V2). Adapted from Stephens, 1991Stephens, , 1993 Associative Learning in Insects From Macphail's (1982) justifiably cautious review of vertebrate learning research, in which the absence of appropriately controlled experiments forced him to cast doubt on the learning capacities of some fish and amphibians, to later assumptions that certainly all vertebrates can learn, research with insects has followed a highly similar course.
In our 2011 paper , we described a handful of studies meant to illustrate the diverse ways in which social, eusocial, and solitary-living insects rely on associative learning. Some of those studies involve Hymenopteran species, an order that includes honeybees and frequently is studied by both learning researchers (e.g., Blaser, Couvillon, & Bitterman, 2006, 2008Couvillon, Hsiung, Cooke, & Bitterman, 2005) and neuroscientists (e.g., Arenas, Giurfa, Farina, & Sandoz, 2009;Lachnit, Giurfa, & Menzel, 2004  Insects are single class of organisms within the superclass Hexapoda (from the Greek, literally "six feet"). The insect orders listed here, as well as the estimated number of families and species in that order, are those described by Resh and Cardé (2009). In this table, we have attempted to name all families in which the associative learning capacity of at least one species has been studied; rows in which no family name is listed are those in which, to our best knowledge, no one has claimed to demonstrate associative learning (with appropriate associative controls) in any member of that insect order. For example, associative learning has not been demonstrated in bristletails, Archaeognatha, the first order listed above. This table is not intended to be an exhaustive list of all papers reporting associative learning in insects. We provide no more than two references for any single insect species, and no more than a few species within a single family. Nonetheless, the figure is designed to expand the information provided in the text, illustrating both the diversity of associative learning in the Class Insecta, as well as obvious gaps in our knowledge.
However, other studies reveal associative learning capabilities in many more diverse members of Hymenoptera, reinforcing the view of widespread associative capacities. For example:  Aphid parasitoid wasps, Aphidus ervi Haliday, are able to retain a preference for a particular odor that they learned as larvae, exhibiting that odor preference much later as adults when, under natural conditions, they would be searching for a host to deposit their eggs (Gutiérrez-Ibáñez, Villagra, & Niemeyer, 2007). A carefully controlled, additional study by Gutiérrez-Ibáñez et al. demonstrates unequivocally that this preference is learned as larvae and retained, rather than due to some form of chemical residue that the larvae carry over to adulthood, as suggested by Corbet (1985).
 Another parasitoid wasp, the tiny Microplitis croceipes (Order Hymenoptera), not only can use learned odor cues to locate food at the astonishing sensitivity of four parts per billion (Rains, Utley, & Lewis, 2006), but also can learn to discriminate between food and host cues under food deprivation and then transfer this learning to situations in which they find themselves well-fed but in need of a host (Lewis & Takasu, 1990).
 Biosteres arisanus (Order Hymenoptera), yet another parasitoid wasp, also shows long-term retention of learned host cues, an ability that gives them, not only an increased ability to locate and parasitize host eggs, but also to produce more offspring capable of reaching adulthood (Dukas & Duan, 2000).
 In a study designed and conducted by the 8-10-year-olds of Blackawton Public School, bumblebees, Bombus terrestris (Order Hymenoptera), demonstrated that they were able to use very complex color patterns -e.g., a 4x4 block paper matrix of two colors, in which only a single spatial arrangement of the two colors was correct -to guide their search for food (Blackawton et al., 2011). Clearly, bumblebees can learn to attend to a more complex arrangement of cues than simple color alone.
Although hymenopterans are one of the most well-studied orders of insects in investigations of associative learning (perhaps surprisingly, as the order Hymenoptera does not contain the largest number either of families or species), several other orders are well represented, as Table 1  and Aedes aegypti, a dengue and yellow fever vector (Menda et al., 2013;Vinauger, Lutz, & Rifell, 2014), are capable of associating learned cues with blood meals and, in at least the case of A. gambidae, can retain this information for at least 72 hours (Chilaka, Perkins, & Tripet, 2012).
 Two insects from the order Orthoptera, namely grasshoppers, Schistocerca americana and locusts, Locusta migratoria, are able to make nutritious food choices based on learned cues. When grasshoppers were forced to search for food that differed in nutritional quality, but were able to rely on learned cues to guide that search, they grew faster than control subjects, exhibited both higher fat and higher non-fat mass, and pupated sooner into adulthood (Dukas & Bernays, 2000). Similarly, locusts that suffered either from a carbohydrate or a protein deficiency were able to use color cues to guide them to the food source previously paired with a diet that contained the specific nutrient they needed (Raubenheimer & Tucker, 1997). Locusts also are capable of using learned cues to locate drinking water (Raubenheimer & Blackshaw, 1994) and to avoid poisons (Lee & Bernays, 1990), yet other ways in which learning guides their nutritional needs. Finally, locusts are able to retain a previously learned response for up to three days and, even more dramatically, the learned response is preserved through metamorphosis from the hopper to adult stage (Goldsmith, Hepburn, & Mitchell, 1978).
 Similar to locusts' learned discriminations, cockroaches, Periplaneta americana (Order Blattodea), are able to use learned cues to find food and avoid poisons (Wantanabe, Kobayashi, Sakura, Matsumoto, & Mizunami, 2003;Watanabe & Mizunami, 2005). Moreover, they can transfer this learning across very different environments and retain the discrimination for at least four days.
 In a demonstration of another long retention test, moths, Cydia pomonella (Order Lepidoptera), were able to form a learned aversion to their preferred host fruit as first instar larvae and still avoid the fruit after they had molted and emerged as second instar larvae (Pszczolkowski & Brown, 2005).
 An important newcomer to the list is Rhodnius prolixus (Order Hemiptera), a blood-sucking insect of Latin America and vector of Chagas disease. R. prolixus, sometimes called the kissing bug because it feeds on people's faces, is one of only two hemipterans that have been studied thus far, despite the fact that Hemiptera, the order of true bugs, is one of the largest, with 104 families and 55,000 different species. Like many of the other species mentioned above, R. prolixus is capable of both appetitive and aversive associative conditioning (Vinauger, Buratti, & Lazzari, 2011a, b;Vinauger, Lallement, & Lazzari, 2013).
 Finally, larval antlions, (Myrmeleontidae spp., Order Neuroptera) are worth mentioning because they don't match the profile of any of the other insects studied thus far. That is, unlike all the other insects used in associative learning experiments to date, antlions are sit-and-wait predators (see Figure 3) that dig conical pits in sandy soil and wait for food, in the form of other small crawling insects, to stumble into the pit trap . Although antlions do not engage in any form of active search (or avoidance), behavior that once thought to be an important indicator of which insect species would be expected to possess the capacity for learning (e.g., Bernays, 1993), and although they possess elaborate sensory structures that enable them to detect their prey well in advance (Mencinger, 1998;Mencinger-Vračko & Devetak, 2008), antlions are capable of using learned cues to prepare for the arrival of their prey. This ability to anticipate prey arrival increases extraction rate and extraction efficiency, which in turn enables them to grow faster and pupate into adults sooner, thus shortening their vulnerable larval stage and reaching the reproductive stage sooner (Guillette, Hollis, & Markarian, 2009;Hollis, Cogswell, Snyder, Guillette, & Nowbahari, 2011;see Hollis, Harrsch, & Nowbahari, 2015, for a review).

A Neuroscientific Conundrum
As research with antlions -and, indeed, as the broad landscape of associative learning research across insects -reveals, a learning profile, a list of behavioral characteristics that can be used to predict which insect species should, and should not, be expected to be capable of learning (e.g., Bernays, 1993), is obsolete. Learning no longer is confined to insects with a comparatively long life, such as those insects that overwinter, or to those that actively search their environment. Moreover, although researchers occasionally dispute the findings obtained from particular species, arguing quite reasonably that the appropriate learning controls were not included (e.g., Alonso & Schuck-Paim, 2006), no sooner does one study become disputed than another study appears, a study with either the same or a closely related species, and with all of the appropriate controls.

Figure 2. A photographic montage of pit-digging antlions (Myrmeleon spp.).
Close-up view of a pit-building larval antlion exposed on the sand surface (top left) and in the process of burying itself under the sand (top right). Bottom: Typical funnel-shaped antlion pits in sand. The winding furrows on the right side of the photograph are the characteristic tracks made by antlions as they search for a suitable pit location; these tracks give rise to antlions' common name, doodlebugs. Adapted from  What, then, should we make of this ever-growing list of insects capable of learning? Dukas (2008) suggests that learning may be universal in all animals with a nervous system. Raine agrees (2009), asserting that learning requires little more than "a sense organ and a simple neural circuit with a switch (which can be reinforced)." (p. R488). Neuroscientists and molecular biologists, as well as many others who study neuronal architecture and development, certainly would agree but would take this assertion a step farther. For example, Greenspan (2007, p. 649) argues that learning emerges as a "fundamental principle of brain functionality," which has been identified recently in Drosophila (Cassenaer & Laurent, 2012;Perry & Barron, 2013), honeybees (Menzel, 2014) and locusts (Holy, 2012). This idea that brain architecture is inextricably linked to learning and memory is not a new one. Indeed, Ramon y Cajal hypothesized that the same processes involved in the development of nervous systems persist into adulthood where they mediate the formation and maintenance of learning and memory, a prescient idea at the turn of the century but one that has been gaining more and more evidence (Carew, Menzel, & Shatz, 1998). Reflecting this same proposition but at a genetic level are demonstrations that the genes involved in circuit refinement during development are those that also are involved in learning and memory in the adult brain (e.g., Leslie & Nedivi, 2011;Rashid, Cole, & Josselyn, 2014). Finally, associative learning is not even necessarily "confined to neural systems" (Fernando et al., 2009, p. 463). That is, molecular circuits for associative learning have been identified in singlecelled organisms (Fernando et al., 2009), as well as in single cells (Kandel & Hawkins, 1992).
Indeed, Brembs, Lorenzetti, Reyes, Baxter, and Byrne (2002) have demonstrated that they could produce biophysical changes in an isolated cell of Aplysia, a single B51 cell, that matched exactly those changes found in operantly trained animals.

Reconciling Neuroscience with Models for the Evolution of Learning: A Proposal
In the context of assertions that learning is an emergent property of neural circuitry and neural mechanisms, and is demonstrated in an ever-expanding list of organisms (see also Mery, 2013), how do we make sense, then, of theories and models that attempt to explain or predict the evolution of learning, implying, as this phrasing suggests, that learning emerges only when particular environmental conditions are met, and that possession of a nervous system is not a sufficient condition in itself? Although the thrust of these models is the evolution of learning qua learning, one possibility is that they actually are exploring the conditions under which plasticity is overridden. That is, although models such as those proposed by Mery and Kawecki (2002) and Dunlap and Stephens (2009) demonstrate quite convincingly how varying amounts of environmental predictability produce the capacity either for inflexible or for learned behavioral responses, testing of the models' predictions necessarily involves extant species. How, then, can we be certain which of the two phenotypes is the default condition? Of course, the view of learning as an emergent property of neural circuits does not change Dunlap and Stephen's (2009) conclusion that "some types of environment change favor learning while others select against it" (p. 3201); rather, what changes is the current emphasis on the "evolution of learning" rather than, we propose instead, on the "evolution of behavioral inflexibility." But this way of reconciling neuroscientific findings with models for the evolution of learning must deal with two issues. One issue is the claim that learning involves costs (e.g., Johnston, 1982;see Burns, Foucaud, & Mery, 2010, for a review). Although Snell-Rood and Papaj (2009) have demonstrated in cabbage white butterflies, Pieris rapae, that whatever the costs are, they are likely to be so low that, even in an environment in which inflexible responses are favored, the ability to learn could be maintained. Nonetheless, other research supports the idea that learning carries costs (Raine, 2009;see Burns, Foucaud & Mery, 2010, for a review). Our answer is that these supposed costs are not those of simple associative learning, per se, which emerges ipso facto from simple neural circuitry. Rather, they are costs associated with cognitive abilities that are based on, but likely surpass, the capacity for simple associative learning possessed by the insects in these studies. One potential (albeit controversial) candidate is observational learning, a form of social learning in which individuals copy the behavior of conspecific demonstrator models. Observational learning has been observed in several insect species, namely flower choice in foraging bumblebees (Leadbeater & Chittka, 2005, 2008, 2009Worden & Papaj, 2005), both oviposition site selection and mate choice selection in female fruit flies (Battesti, Moreno, Joly, & Mery, 2012;and, Mery et al., 2009, respectively), and antipredator behavior in crickets (Coolen, Dangles, & Casas, 2005). Although this ability to copy the behavior of another individual is thought to originate in simple associative learning (Giurfa, 2012;Dawson, Avarguès-Weber, Chittka, & Leadbeater, 2013;Giurfa, 2012;Heyes, 2012), an additional cognitive module might be required that enables the organism to attend to the social stimulus and the contingency that the social stimulus is experiencing.
A second issue is this: If all animals, vertebrates and invertebrates alike, possess both fixed and learned responses, what does it matter whether evolutionary models adopt the learning or non-learning default condition? The distinction is subtle but, we argue, important. By recognizing that nervous systems are built to form associations, we necessarily shift our perspective, focusing on a very different set of questions. For example, we begin to look for a profile of species, or of behavioral systems within a species, that overrides, rather than promotes, learning: What species do not use associative learning in any aspect of their search, say, for food, and what is it about their behavioral ecology that is similar to other species that do not need to learn the characteristics of rivals, predators or mates? Can we begin to construct an ecological profile of what those non-learning behavioral systems look like? Are they similar in both vertebrates and invertebrates? This kind of approach already has been adopted, very successfully, by neuroscientists who study the way in which the brain disables costly memory under food shortage in Drosophila (Plaçais & Preat, 2013), or the way in which desert locusts' ability to form learned aversions is blocked when population densities increase (Simões, Niven, & Ott, 2013). This shift in perspective is critically important: Only by asking the right questions can we begin to understand the evolution of behavior.