Higher-temperature phases of a structured neural model of cortex

The trion represents a localized group of neurons with three levels of firing activity. Networks of a small number of trions (with structured interactions and firing dependent on activity at two previous discrete time steps) support a repertoire of hundreds of quasistable, periodic firing patterns (which can be learned). We report striking phenomena with variation in "temperature" T: There exist a series of "phase transitions" at precise values T(n), giving new repertoires of periodic firing patterns, and the average time for any initial firing configuration to project onto a firing pattern shows a quite sharp change at each T(n). Near a phase transition, in a Monte Carlo simulation, the temporal evolution wanders back and forth between sets of these firing patterns in contrast to the more structured sequential evolutions far from a T (n).

exist a series of "phase transitions" at precise values T(n), giving new repertoires of periodic firing patterns, and the average time for any initial firing configuration to project onto a firing pattern shows a quite sharp change at each T(n).Near a phase transition, in a Monte Carlo simulation, the temporal evolution wanders back and forth between sets of these firing patterns in contrast to the more structured sequential evolutions far from a T (n).
One of the important problems in science is the under- standing of information processing and memory in the brain.Theoretical studies of neural networks have been addressing these problems for over 50 years.The formal mathematical application of the powerful physical spin system formalism to neural networks began with the seminal paper by Little.' The temperature T and the Boltzmann relation for the probability of neuron firing in Little's Ising spin model were shown by Shaw and Vasudevan to follow from the known statistical Auctuta-  tions of neurotransmitter release at the synapse.
Modifications of Little's model by Hopfield has led to the popular spin-glass neural network analogy, which is characterized by long-range random (or by zero strength) connections between neurons (before learning) and asynchronous updating of neuronal firing.An opposite ap- proach by us was to modify Little's model in a direc- tion inspired by Mountcastle's organizational principle for cortical function and by the striking axial next- nearest-neighbor Ising (ANNNI) model results of Fisher and Selke.
(i) Following Mountcastle, the resulting trion model considers the basic neural network to be the well-established cortical column (about 700 pm in di- ameter) comprised of small irreducible processing sub- units (trions).This column or network of subunits has the capability of being excited into comp1ex spatial- temporal firing patterns.The creation and transformation of such patterns constitute the basic events of short-term memory and information processing.We as- sume that higher mammalian cortical processes involve complex spatial-temporal patterns defining the code [this is in contrast to models (e.g., the early spin-glass ' mod-  el) that assume that this code only involves sets of neu- rons firing with high frequency].It is argued that the appropriate spatial scale of these subunits is about 100 pm in diameter (about 100 neurons), and that there exists a temporal scale or time step ~of about 50 ms for these groups of neurons to "burst" in synchrony.(ii) Following Fisher, the interactions among trions are taken to be lo- calized, competing (between excitation and inhibition) and highly structured, and the firing state of the system is updated in a probabilistic way related to the states at the two previous time steps.We have found that the trion model combines the computational power of cellular au- tomata' with the learning, adaptive capabilities of the standard neural models." A small trion network has a repertoire of hundreds of quasistable, periodic, spatial- temporal firing patterns defined as magic patterns or MP's.The MP's have the striking properties of having high probabilities of cycling through the entire pattern, and can be "learned" with only small modifications in the interaction strengths using a Hebb-type algorithm.' The MP's evolve temporally in Monte Carlo simulations in certain natural sequences from one to another, as in Fig. 1 of Ref. 5.
The purpose of this paper is to present striking results that are of strong physical interest concerning the nature of the many "phase transitions" that occur at precise values [see Eq. ( 4)] of higher temperature T(n), giving new repertoires of MP's.In the example given below (Fig. 2), the repertoire in the second phase region has an order of magnitude more MP's than the lowest T phase region.The average time (t) for any initial network firing configuration to "recall" an MP has a sharp change at each T(n).Near a T(n), in a Monte Carlo simulation, the temporal evolution wanders back and forth between sites of MP's, in contrast to the more structured sequen- tial evolutions far from a T(n).
The probability P;(S) of the ith trion having a firing level or state S at time n ~is given by P; (S)=g (S)exp(BM, S) g g (s)exp(BM, s), S M; = g ( V, S, '+ 8;)S, " . ) -V, , J 43 5678 where S' and S" .are the states of the jth trion at the two earlier times (n -1)r and (n -2)r, respectively, as shown in Fig. 1.V, and 8'.
~are the interactions between trions i and j at time r from times (n -1)r and (n -2)r, respec- tively; V, is an e6'ective firing threshold.The three possi- ble firing states (of each trion) denoted by +, 0, and- for S =1, 0, -1 represent, respectively, a large "burst" of firing, an average burst, and a below average firing.
The statistical weighting term g (S) with g (0) ))g (+, -) takes into account the number of equivalent firing configurations of the trion's internal neuronal constitu- ents.The fluctuation parameter B is inversely propor- tional to the noise or "temperature, " and was explicitly related to the statistical nature of neurotransmit ter release at the synapse: we define the temperature T = B '. Learning a pattern takes place through the Hebb-type algorithm' b. V~= @ g S;(n~)S.((n -1)r), bW~=@AS;(nr)SJ((n -2)r), e) 0, (2) which allow for "experience" (via the parameter e) to modify the interactions between trions.Equations ( 1) and ( 2) completely describe the trion model.
For this paper, we consider a one-dimensional ring of N trions.For a given set of V 's, W ' 's, g (S), and B we ex- amine all 3' ' possible firing configurations of the first two time steps.The computer is instructed to search for all the quasistable (high probability of cycling) periodic firing patterns, defined as MP's.The MP's are found by computing the most probable temporal evolution of the trion states using (1) starting from each of the 3' ' initial network firing configurations (S'&', . . ., Sg, S'i, . . ., S&) and determining if that evolution leads to a pattern that repeats, an MP.This program also determines the aver- age number of time steps (t ) for the set of initial firing configurations to recall an MP.Small networks with highly structured, competing (between excitation and in- hibition) interactions are found to have huge repertoires of MP's that will temporally evolve in Monte Carlo cal- culations from one to another (see Fig. 1 of Ref. 5).Ex- perience or learning can select out (as in the model of Edelman' ) MP's with only small changes in interaction strengths using (2).Many initial firing configurations project onto or recall an MP.This recall is extremely rapid: the average number of time steps ( t ) is 2 to 4.
The statistical fluctuations are crucial for the full richness of the trion model.This we can show from the fol- lowing simple analysis: In the deterministic limit (B go- ing to infinity, or T to 0) and with any small deviation of the interactions from a precise structure, an MP having an S =0 level would have vanishing probability of cycling P, (B).We see (e.g. , in Fig. 4 of Ref. 6) that P, (B) van- ishes both at large B and at small B. Thus finite fluctua- tions are necessary to stabilize the high firing probabili- ties of the MP's that contain many S =0 levels.Now we present a new phenomenon not at all related to the T =0 limit: There is a series of transition temperatures T ( n ) between which new sets of repertoires of MP's exist, and   sharp changes in (t ) occur at these T(n).This is a counter example to the conjecture' that cellular automa- ta with explicit statistical fluctuations McGRANN, SHA%, SILVERMAN, AND PEARSON g(0)/g(+, -)=500 or u =ln(500)=6.2146, V;;+i = 1, 8'" = -V; V;;=2, with V; and all other V; and 8'" equal to zero, and ring- like connections (trion i =trion i +1V).This nearest- neighbor example will have eight positive integers ~M;w ith values n =1 to 8 depending on the network firing levels S" and S' at the two previous time steps.Thus we will have a different competing logic for the eight values of T(n): T '(n)=B(n)=6.2146/n, n =1, . . ., 8 .
These values (4) will correspond exactly to "transition temperatures" T(n) separating regions with difFerent re- pertoires of MP's, and having sharp changes of ( t ) at these T(n).This is illustrated in Fig. 2 for the %=6 trion network.Two sets of calculations are presented, one with a 0.1% random deviation for each trion's con- nections away from the precise values (3), and one with 30 times the same deviations (3%).

IG. 2. (a)
The number of MP's vs (inverse) temperature T =B for two sets of calculations .o a -1 . r an N = 6 trion network with con- nections (3) modified by a 0.1% random deviation and a 3% random deviation.[These calculations were done by following the sin (1).~jThe 0.1% deviation curve has the sharper transitions.Only the n =1-3 transitions (4) most-probable-path time evolution using . & e .o evia ion c this scale.The n =4 transition corresponds to the repertoire changing from two MP's to t e sing e a rions in can be seen on t is sca e.
e n = rans h S =0 level).Each distin uishable spatially rotated pattern is counted as a distinct MP, , imi ar o a or e of time steps (t ) for an initial firing configuration to project into an MP.All eight transitions, , in p creases in (t j occur or n = -.c n en argem ( ) f =1-3.( ) A largement of the T ' vs (t) curve in (b) at the n =2 transition.The sharp peak in the curve corresponding to 0.1% random deviations from connections ( ) i q ' g. ' it ions (3) is uite strikin .[A further calculation (not shown) wit 0.01% deviations demonstrated that the peak was a plateau of height 3.S5 and width proportional to this percentage deviation.
An enlargement of the low T ' vs (t ) region in (b) to clearly show the four transitions n =5-8 not present in (a). the transitions.Figure 2(a) shows the number of MP's   versus T '=8; note the huge increase in the repertoire going across the T(n = 1) transition.Other transitions in Fig. 2(a) are seen at the n =2 and 3 T(n) values; a further transition not seen on this scale occurs for n =4 to the single MP with all trions in the S =0 level.We see in Fig. 2(b) that all eight transitions are present in the plots of (t ) versus T ', i.e. , the n =5 -8 transitions are only seen in (t ).Sharp spikes in (t ) occur for n =1-3; a blowup of the n = 2 region in Fig. 2(c) clearly demon- strates this "slowing down" at the phase transition.A further set calculations using O. l%%uo deviations showed that the "slowing down" at T(n =2) was a plateau of height 3.5 and width proportional to the precentage ran- dom deviation used for the interactions away from the values (3).To illustrate the nature of the MP's, we show in Fig. 3 all the MP s from the repertoire for the six-trion network with connections (3) from the phase region below T(n =1).To appreciate the magnitude of the re- pertoire of MP's from our structured connections, con- sider a seven-trion network with connections (3) from the phase region between T(n =1) and T(n =2).This re- pertoire has 4496 MP's.In contrast, a similar network randomly connected had a repertoire of 30 MP's along with a very large increase in ( t ).
-.::::: :r.'::::::i::: The shadings white, gray, and black correspond to S =1, 0, and -1, respectively.Time in each MP is represented as advancing downward.These include all distinct spatial rotations and can be completely classified by the 34 MP's shown here that cannot be rotated into each other.For the connections (3), all the MP's have cycle length 6 (except for the one with all S =0); other choices of connections (Ref.6) yield diAerent cycle lengths.In a calculation of the most probable pathway, once the network is in an MP, the MP will continue to cycle repeatedly.However, in a Monte Carlo calculation (see Fig. 4), transitions occur from one MP to another.

FIG. 4. A Monte
Carlo calculation where the connections (Ref.3) were modified asymmetrically so that V;; & = 0. 8, W;;+, = -0.9, and then a 10% random deviation was added.The inverse temperature T '=3.107, which is the n =2 transi- tion for the symmetrical connections.Notice the Aow between MP's as the network evolves in time starting with the first column and ending downward and then up to the top of the second column, etc.A Monte Carlo evolution with temperature T far from a phase transition quickly evolves through a struc- tured sequence of MP's into one of the "dominant" MP's, in which it remains usually by the equivalent of 50 time steps.The MP at the top left of Fig. 3 is the "dominant" one for T (T(n =2), whereas the MP in the bottom right of Fig. 3 is the "dominant" one for T ) T(n =2).for an N =6 trion network with connections the same as those for the Monte Carlo calculation in Fig. 4. The N =1 and N =2 transitions are smeared out considerably with the total number of patterns reduced, while the other transitions are not apparent [see Fig. 2(a) j. from Little's neural network Ising spin analogy, ' and from the Fisher and Selke ANNNI model.As is strik- ingly demonstrated in Fig. 2, we see "phase transition" phenomena in the trion model, which we believe are of strong interest to the physicist interested in (a) cellular automata' '' as well as in (b) problems of "critical" be- havior of structured systems: ' (a) As noted above, the series of transition temperatures T(n) between which new sets of MP's exist, and the sharp changes in (t ) which occur at these T(n), provide a counter example to the conjecture' that cellular automata with explicit sta- tistical fluctuations have features equivalent to those found in deterministic models.(b) The series of higher- temperature transitions at T(n) are analogous to the higher T phase boundaries found in the ANNNI model by mean-field-theory calculations.' We also suggest that these phenomena might be studied using the recent memory in Lie algebras (MILA) model' based on zero- level representations of Kac-Moody algebras' to describe adaptive temporal development of structured finite sys- tems.The relevance of our results to neurobiological ex- periments will be presented elsewhere.'   the transitions, yet maintaining a sizable part of the MP repertoire (see Fig. 5) from the unbroken case (3).In the Monte Carlo case shown in Fig. 4, we observe the Aow back and forth among MP's.

The trion model of cortical organization was developed
This work was supported in part by the Research Em- ployment for Undergraduates Program from the Nation- al Science Foundation.We thank K. Shenoy for help with computer programming.W. A. Little, Math.Biosci.19, 101 (1974); also see B. G. Cragg and H. N. V. Temperley, EEG Clin. Neurophys. 6, 85 (1954), for the concept of the magnetic analogy.
~exp(u S .+BM,S).The full richness of the trion model occurs due to the competing of the two terms in the exponential that allow the S =0 firing level to com- pete with the S =1 and -1 levels for finite M. Consider a specific example.%'e will choose (. 1.Schematic diagram of a ringlike network of N trions (trion i =trion i +N) at three time steps showing the states and con- nectivity.
FIG.5.The number of MP's vs (inverse) temperature T model see W. A. Little and G.