Brain Evolution, Development, and Plasticity

Rayna M. Harris, Lauren A. O’Connell, and Hans A. Hofmann

15.1 Brain Evolution and Development

Across animals there is astonishing diversity in the structure and function of nervous systems and the resulting behavior patterns. Not surprisingly, the question of how this diversity has evolved has long fascinated biologists, prompted initially by the observation that allometric relationships exist between the size of the brain—or brain region—and body size across a wide range of vertebrates (Striedter, 2005). Yet it was not until fairly recently that the mechanisms that make such variation possible have become a focus of study. Brain development and plasticity are clearly dynamic processes that change neural structure and function on a variety of time scales, from early patterning of the developing brain and neural changes within an individual’s lifetime to changes over evolutionary time. In the present chapter we discuss brain development and plasticity across levels of neural organization in a comparative framework to shed light on brain evolution across and within vertebrates and, to a lesser extent, invertebrates.

With the exception of sponges and placozoans, all animals have a nervous system. During the course of evolution, nervous systems in diverse taxa showed increasing cephalization and regionalization. Cephalization refers to the tendency for nerve cells to concentrate near sensory organs (i.e., mouth, eyes, nose) at the front end of the body. Regionalization refers to the idea that specific brain areas carry out specific functions. These organizational principles are accompanied by an ever increasing complexity in the diversity of neuronal cell types, functions, and connections (Striedter, 2005).

Properties of the environment are often thought to dictate which physical and sensory adaptations will be successful, and much research has focused on how socioecological pressures sculpt brains throughout evolution (Pollen & Hofmann, 2008). There is a strong positive correlation between brain size and body size, a phenomenon known as allometry (Snell, 1892; Thompson, 2011). In order to facilitate more robust comparisons across taxa the resulting allometric scaling exponent (usually ranging between 0.67 and 0.75) can be used to calculate an encephalization quotient for each species, which can be highly variable across vertebrate species and has often been associated with cognitive abilities within the context of comparative analyses (Harvey & Krebs, 1990; Jerison, 1973). Mammals and birds are in the upper portion of this range, and their position is often attributed to the increased size and complexity of the cerebral cortex in terms of the number of layers and neurons. The increase is a continuum, and correlations between large cortex and complex social behavior are very strong. However, is cortical expansion really responsible for increases in cognitive and behavioral complexity? Comparative studies have provided insight into which socioecological variables best explain variation in phenotype (in this case brain size) across populations and species (Pollen et al., 2007; Pollen & Hofmann, 2008).

Two models have been proposed to explain how brains evolve: the adaptationist model (often also referred to as “mosaic evolution”) and the developmental constraints model (Pollen & Hofmann, 2008).

The adaptationist model suggests that the brain contains functionally distinct regions (or modules) that mediate particular sets of behaviors (Barton & Harvey, 2000). Selection on a specific set of behaviors should favor a change localized to the brain region mediating that behavior. A few studies have provided support for this model. For example, Barton and Harvey (2000) showed that structure size correlates with functionally related structures in both primates and insectivores. Wang Mitra and Clark (2002) found that the fraction of the adult brain occupied by the telencephalon is significantly larger in socially complex birds, while eating habits, migration patterns, mating type, and vocal learning did not correlate with telencephalic fraction. Reader and Laland (2002) also found that telencephalon size is correlated with innovation frequency and social learning in primates. However, it is important to understand that causal relations are not always clear in these and other studies, and even though these adaptive hypotheses may be plausible, they are difficult to test.

On the other hand, the developmental constraints model recognizes that a common set of genes and developmental processes may regulate the development of a range of functional regions. Finlay and Darlington argue that developmental timing can explain much of the variation in brain structure size. In their model of brain size evolution, they argue that selection for a change in any single brain structure would cause the brain to change as a whole unit (Finlay & Darlington, 1995; Finlay, Darlington, & Nicastro, 2001). They find evidence that brain structure sizes across mammals are strongly correlated with the brain size according to different power relationships, such that the neocortex exponent might explain higher neocortex fraction in primates. The authors posit that shifts in the developmental time of cortical neurogenesis between primates and rodents explain the expansion of the neocortex in primates (Finlay & Darlington, 1995). A synthesis by Striedter (Striedter, 2005) provides support for both models, suggesting that both mosaic evolution and developmental constraints play fundamental roles in driving brain/behavior changes (see also Chapter 13, this volume).

It is important to keep in mind that there are several potential confounds that often make the interpretation of comparative studies susceptible to simplistic adaptationist interpretations (Gould & Lewontin, 1979; Pollen & Hofmann, 2008). First, we usually do not know the selective forces that were at work during a given period of evolution. Second, genetic drift instead of selection can cause changes in neural and behavioral phenotypes. Third, because of their common evolutionary history, traits across species within a hierarchical and branched phylogeny cannot be considered independent, and therefore, in order to draw conclusions from the covariation of traits across taxa, this phylogenetic nonindependence needs to be taken into account (Felsenstein, 1985; Harvey & Pagel, 1991; Pagel, 1999). Since Felsenstein’s classic paper, the generally accepted method of overcoming the effect of shared ancestry has been to calculate differences in (extant and ancestral) trait values between sister taxa. Two traits are then considered evolutionarily correlated (i.e., change in one trait has been accompanied by change in the other) if these (standardized) differences—or phylogenetically independent contrasts—in one trait significantly covary with contrasts in the other trait (Garland, Harvey, & Ives, 1992). Even though more sophisticated approaches have since been developed (Freckleton, Harvey, & Pagel, 2002; Pagel & Meade, 2006), the fundamental assumption is that the phylogenetic relationships between the species studied are known. However, even for groups that have been relatively well studied, well‐resolved phylogenies often do not exist, and it is of paramount importance to conduct comparative analyses for the different phylogenetic hypotheses if a consensus has not yet been reached.

15.2 Developing Diverse Brains

How can we explain the diversity of the structures that make up vertebrate brains? Beyond the “just so” stories that often characterize the interpretation of the causes and origins of brain diversity (Healy & Rowe, 2007), two problems have vexed this line of research. First, it is not at all obvious how an increase in (relative) size would give rise to functional differences (e.g., increased cognitive abilities, novel sensory specializations, or behavioral complexity). Although a larger number of neurons and/or synapses might well result in greater processing power and/or speed, there is no clear relationship between such measures and behavioral or cognitive outcomes. Second, our understanding of the developmental mechanisms that give rise to the observed variation in brain structure is still very limited. In this context it is also important to keep in mind that differences in brain structure and function can be as much a consequence of environmentally responsive developmental plasticity as of genetically driven developmental control (Pollen & Hofmann, 2008).

15.2.1 Generating Diversity through Early Patterning

Many studies have suggested that neurogenesis later in development generates diversity, which might result in the differential expansion of various brain areas (see below). Similar to the basic patterning processes that specify the main body axes across all metazoans, the overall spatial and temporal activity patterns of transcription factor networks that establish the main compartments during early brain development are highly conserved (Puelles, Harrison, Paxinos, & Watson, 2013; Puelles & Rubenstein, 2003). This neuromeric model describes the spatiotemporal patterns of highly conserved developmental genes, which divide the developing brain into anteroposterior segments (neuromers) prefiguring adult functional units, and uses this information to identify homologous structures across species (Puelles et al., 2013). Because the genomic control of neural morphogenesis is remarkably conservative, the relationship between embryonic patterns and adult structure is very consistent across vertebrates (see Chapter 12 this volume). This developmental framework has thus been key to resolving putative homology relationships across vertebrates for numerous brain regions (O’Connell & Hofmann, 2011a). However, it should also be noted that many homologies are still considered tentative (Goodson & Kingsbury, 2013) and that comparisons across vertebrates that include teleosts continue to be particularly challenging because actinopterygian (ray‐finned fish) forebrains develop via eversion not invagination (Yamamoto et al., 2007).

Given such a conserved theme of brain development, could small variations arising from developmental expression profiles potentially result in substantial, and possibly adaptive, changes in brain structure? This question has received surprisingly little attention. Insights into the developmental processes that give rise to brain diversity can be gained by examining the remarkable phenotypic diversity found in the cichlid fishes from East Africa’s Great Lakes, which have undergone the most rapid and extensive adaptive radiations known for vertebrates. They display an astonishing array of phenotypes with little genetic diversification (Renn, Aubin‐Horth, & Hofmann, 2004). The extraordinary ecological (e.g., habitat, feeding specialization) and behavioral (e.g., color preferences by females, mating and parental care systems) diversity is correlated with variation in brain structure of a magnitude that exceeds that of all mammals and facilitates comparisons across large social and physical gradients in closely related species of cichlids (Pollen et al., 2007).

In an elegant study in cichlid fishes from Lake Malawi, Sylvester et al. (2010) examined gene expression variation in a regulatory circuit (composed of six3, fezf2, shh, irx1b, and wnt1) known to specify anterior‐posterior brain polarity and to set the boundary limits between the developing fore‐ and midbrain. There is considerable variation in the expression patterns of these genes between rock‐dwelling mbuna (Labeotropheus fuelleborni, Maylandia zebra, and Cynotilapia afra) and sand‐dwelling nonmbuna cichlids (Copadichromis borleyi, Mchenga conophorus, and Aulonocara jacobfreibergi), consistent with the differences observed in the relative size of fore‐ and midbrain structures in adult fish. When the WNT signaling pathway is chemically perturbed in the developing embryo, alterations in this coexpression network are sufficient to give rise to the observed differences in brain development, resulting for instance in a rock‐dweller with the forebrain shaped and sized like that of a sand‐dweller. These results strongly suggest that evolutionary changes in the patterning of developing brain compartments can establish ecologically and behaviorally relevant differences in the adult brain. Variation in subsequent neurogenesis, which until now has been thought to be the main source of variation in brain structure across species, may then elaborate the construction of diverse brains (Sylvester et al., 2010). Clearly, diversity in early patterning constitutes a potentially important, yet hitherto underappreciated, avenue by which natural selection can act on brain structure and function, possibly releasing the brain to some extent from developmental constraints imposed by cell proliferation mechanisms common across brain regions.

15.2.2 Neuronal Cell Fate and Development

Our understanding of neural development and brain function in part depends on an understanding of the cell fate of a neuron and its location and connectivity in the brain. To illustrate the role of this information in comparative brain development and plasticity we discuss two examples: the specification of dopaminergic neurons in the brain and the caudal migration of gonadotropin‐releasing hormone (GnRH) neurons early during development.

Dopamine is an ancient neurochemical that, in diverse species, modulates the selection of behavior patterns such as basic motor programs (Joshua, Adler, & Bergman, 2009; Vidal‐Gadea et al., 2011), social behavior (Aragona & Wang, 2009; O’Connell & Hofmann, 2011b), and learning and memory (Hyman, Malenka, & Nestler, 2006; Wise, 2004). In mammals, dopaminergic cell populations are limited to a relatively small number of discrete brain regions, while in teleosts more than 20 groups of dopamine neurons have been described (O’Connell, 2013). How this variation comes about and to which extent it contributes to differences between lineages is not well understood, as these cell populations are not easy to homologize across vertebrates. Nonetheless, gene expression patterns in dopaminergic neurons of the posterior tuberculum are consistent with those of the tetrapod ventral tegmental area (O’Connell, 2013), which releases dopamine into the reward system. Flames and Hobert (2009) proposed a conserved regulatory code that specifies and maintains dopaminergic neurons from Caenorhabditis elegans worms to vertebrates, although a detailed evolutionary understanding of these neurons has remained elusive (reviewed in O’Connell, 2013).

GnRH neurons comprise a small population of neuroendocrine cells in the rostral hypothalamus and basal forebrain where they serve as a key regulator of vertebrate reproduction (Gore, 2002a). Like most peptidergic cell groups, they are born in the olfactory placode early in development but migrate caudally as embryogenesis proceeds. They secrete gonadotropin‐releasing hormone (GnRH‐1), communicate with many areas of the brain, and integrate multiple inputs to control gonad maturation, puberty and sexual behavior. GnRH‐1 neurons migrated from olfactory bulb and midbrain. The exact mechanisms of this migration and target finding are under intense study (Sabado, Barraud, Baker, & Streit, 2012), but cell‐specific molecular profiling has provided increasing evidence that these neurons are part of an ancient class of neurosecretory cells already present in the last common ancestor of all bilaterian animals (Tessmar‐Raible et al., 2007).

15.2.3 Differential Proliferation Dynamics Generate Variation in Cortex Size

The evolutionary expansion of the cerebral cortex in mammals, particularly in primates, has fascinated scientists for some time (Finlay & Darlington, 1995; Reader & Laland, 2002). The increase in cortex size in the lineage leading to humans has been interpreted as the result of variation in neurogenesis later in development, when cells in pre‐established compartments proliferate, die, and/or differentiate into mature neurons and glia cells. According to the radial unit hypothesis, simply altering the first of the three phases of cell division that produce cortical excitatory neurons can scale the size of the cortex (Rakic, 1995). In contrast, the intermediate progenitor hypothesis, which seems to have stronger support, suggests that, in the evolutionary expansion of the cortex, proportionately more neurogenesis occurs during the third and final phase of proliferation (Hill & Walsh, 2005; Kriegstein, Noctor, & Martínez‐Cerdeño, 2006).

Scientists have begun to unravel the molecular mechanisms regulating the size of the neocortex. Given the importance of differential proliferation dynamics in determining cortex size discussed above, it is no surprise that the mitotic spindle protein, ASPM (abnormal spindle‐like microcephaly‐associated protein) is a major player in the process (Pulvers et al., 2010). It is known that mutations in ASPM cause microcephaly (decrease in brain size) in some human families (Bond et al., 2003), and that it has undergone positive selection in the primate lineage leading to humans (hominids) (Kouprina et al., 2004). β‐catenin is another protein that appears to control cerebral cortex size through its effects on cell proliferation during cortex development via Wnt signaling (Chenn & Walsh, 2002). While these and other studies have identified putative genetic events underlying the evolution of the human brain and its emergent cognitive capacities, allelic variation in ASPM or Microcephalin does not seem to be associated with IQ in humans (Mekel‐Bobrov et al., 2007), which again underscores the previous insight that the functional implications of variation in the size of a brain structure are often unclear. Also, we need to ask what the relative importance of differential proliferation is compared with the initial delineation of the future pallial vs. other areas much earlier during development, as discussed above (§15.2.1). Specifically, variation in early patterning might reduce the developmental constraints that otherwise limit the extent to which natural selection can sculpt neural structure and function in a brain‐region‐specific manner.

15.2.4 Cortical Development Is Remarkably Plastic

There is an astonishing degree of diversity in cortical organization across vertebrates (Krubitzer & Dooley, 2013). For example, somatosensory cortical maps reflect biological adaptations. In the naked mole rat, the somatosensory cortex is dominated by the representation of teeth (Catania & Remple, 2002), while in the human it is dominated by the mouth, hands, and eyes (Marieb & Hoehn, 2012). However, cortical development is very plastic, and altering the environment can alter the structure of the brain and thereby possibly its function. One study showed that a considerable portion of the developing cortical sheet could be removed and functional regions that would normally appear in the removed area are accommodated elsewhere (Huffman et al., 1999). Studies on humans and other vertebrates that have undergone limb amputations or sensory organ removal show similarly plastic remodeling of the cortex (Farnè et al., 2002; Karlen & Krubitzer, 2009). Clearly, there is a lot of evolutionary and developmental plasticity, but how does it come about mechanistically?

15.3 Neural Circuits, Neurochemicals, and Behavior

To understand how the brain mediates a behavioral output, it is necessary to understand both the changes in gene expression that occur in response to external or internal stimuli and the neural circuitry in which these changes take place. Here we introduce the neural circuits that govern (social) behavior, how neurochemicals and hormones modulate those circuits, and how these hormone‐neurotransmitter systems have evolved.

15.3.1 Inferring Homologies for Neural Circuits Underlying Social Behavior

Do “complex” behaviors drive the evolution of complex brains? For example, is the size of the primate neocortex a result of high‐quality foraging and Machiavellian social competition, or a simple consequence of body size? Interdisciplinary efforts to combine neuroscience, evolution, and development have given rise to the field of “neuro‐evo‐devo” and have shed light on the evolutionarily conserved neurochemical circuits that underlie behavior. As described in §15.2.1, comparative work across bilaterians has demonstrated how early developmental patterning partitions functional units of the developing brain. These comparative and integrative approaches have facilitated a mechanistic understanding of the evolution of variation in brain morphology, neural phenotypes, and neural networks that determine brain function and give rise to behavioral diversity across taxa (O’Connell, 2013).

All animals evaluate the salience of external stimuli and integrate them with internal physiological information to produce adaptive behavior. Natural and sexual selection impinges on these processes, yet our understanding of behavioral decision‐making mechanisms and their evolution is still very limited. Insights from mammals indicate that two neural circuits are of crucial importance in this context: (1) the social behavior network, consisting of amygdalar and hypothalamic regions that regulate multiple forms of social behavior (sexual behavior, aggression, and parental care), are reciprocally connected, and contain sex steroid hormone receptors (Goodson, 2005; Newman, 1999) and (2) the mesolimbic reward system, which evaluates the salience of an external stimulus and consists mostly of telencephalic brain regions and dopaminergic projections from the midbrain ventral tegmental area (Deco & Rolls, 2005; Wickens, Budd, Hyland, & Arbuthnott, 2007; Wise, 2005). Based on a synthesis of neurochemical, tract‐tracing, developmental, and functional lesion/stimulation studies, O’Connell and Hofmann (2011a) delineated homology relationships for most of the nodes of these two circuits across the five major vertebrate lineages (mammals, birds, reptiles, amphibians, and teleost fish; see Figure 15.1B, D). Even though many of these homologies should still be considered tentative (Goodson & Kingsbury, 2013), this comparative analysis of the two neural circuits clearly suggested that these circuits were already present in early vertebrates and that together they form a larger social decision‐making network that regulates adaptive behavior. This synthesis provides an strong foundation on which we can build research programs to better understand the evolution of the neural mechanisms underlying reward processing and behavioral regulation (O’Connell & Hofmann, 2011a).

Image described by caption. — **Figure 15.1** Levels and Time Scales of Neural and Behavioral Plasticity.

Understanding how genetic and environmental factors affect neural plasticity requires detailed analyses of development, neural networks, neural connectivity, life history, and evolution. (A) Gene expression patterns of developmental genes (colored regions) are highly conserved and regulate development of homologous brain structures.

Adapted from O’Connell 2013.

(B) The social decision making network provides a framework for analyzing structurally and functionally homologous brain regions across vertebrates.

Adapted from O’Connell & Hofmann 2011b.

(C) Life history traits and transitions are highly diverse, but social behavior (e.g., courtship and mating) is present in all kingdoms. Neural activation patterns underlying behavior can be conserved or divergent across species. (D) Phylogenetic relationships between major vertebrate lineages.

15.3.2 Evolution of Neurochemistry Underlying Behavior

Establishing homology across vertebrate brain regions mediating social behavior has opened exciting new opportunities. In particular, it invites study of how variation across taxa in the neural basis of social decision making might explain observed differences in behavior—as well as how and why these differences evolved. A recent comparison of the social decision‐making network across 88 vertebrate species has revealed that, although neurochemical profiles are very much conserved, vertebrate lineages differ more in the spatial distributions of ligands (cell populations that synthesize neuropeptides, neurotransmitters, or steroids) than their receptors (neuropeptide and neurotransmitter receptors, sex steroid hormone receptors) (O’Connell & Hofmann, 2012). It is important to note that this large‐scale comparative study only noted presence or absence of a neurochemical or gene in a particular brain region; however, quantitative variation in neurochemical gene expression, also seems to be important for variation in behavior within lineages. An extensively studied example is how quantitative variation in the vasopressin receptor expression in different species of Microtus voles is linked to differences in mating systems (reviewed in Wang, Young, De Vries, & Insel, 1998).

15.3.3 Neurotransmitter Circuitry Underlying Decision Making

Animals are constantly confronted by challenges and opportunities in their social environment in which they must make adaptive decisions to ultimately maximize their fitness. Before responding to a social stimulus with a behavioral output, animals must first evaluate the salience of a stimulus. The neural circuit in which this evaluation takes place is thought to be the mesolimbic dopamine system (Deco & Rolls, 2005; Wickens et al., 2007; Wise, 2005), with a key role for dopaminergic projections from the ventral tegmental area of the mammalian midbrain to the forebrain. In mammals, dopaminergic projections from the ventral tegmental area to the nucleus accumbens encode both rewarding and aversive stimuli while projections to the prefrontal cortex encode aversive stimuli selectively (Lammel, Ion, Roeper, & Malenka, 2011). The neuroanatomical components of the dopamine reward circuitry seem to be conserved across vertebrates (O’Connell & Hofmann, 2011a), which is not surprising given its crucial role in evaluating stimuli in many species.

Although dopamine is a key neurotransmitter for encoding the value of stimuli in vertebrates, octopamine (homologous to the vertebrate norepinephrine) plays a prominent role in arthropods (Barron, Sovik, & Cornish, 2010). Studies in honeybees show that an individual’s response to a reward (proboscis extension to sucrose) was reduced when injected with dopamine (Mercer & Menzel, 1982), whereas octopamine enhanced the proboscis response and could even substitute for sucrose presentation (Hammer & Menzel, 1998). This study highlights the opposite roles of dopamine and octopamine in arthropod behavior, recently confirmed in pharmacological manipulations of both the cricket Gryllus bimaculatus (Mizunami et al., 2009) and honeybee Apis mellifera (Farooqui, Robinson, Vaessin, & Smith, 2003; Vergoz, Roussel, Sandoz, & Giurfa, 2007) as well as genetic manipulations of the fruit fly Drosophila melanogaster (Schwaerzel et al., 2003), where the octopamine is necessary for reward learning and dopamine is necessary for aversive learning (reviewed in Barron et al., 2010).