Understanding Perception Through Neural “Codes”

A major challenge for cognitive scientists is to deduce and explain the neural mechanisms of the rapid transposition between stimulus energy and recalled memory-between the specific (sensation) and the generic (perception)-in both material and mental aspects. Researchers are attempting three explanations in terms of neural codes. The microscopic code: cellular neurobiologists correlate stimulus properties with the rates and frequencies of trains of action potentials induced by stimuli and carried by topologically organized axons. The mesoscopic code: cognitive scientists formulate symbolic codes in trains of action potentials from feature-detector neurons of phonemes, lines, odorants, vibrations, faces, etc., that object-detector neurons bind into representations of stimuli. The macroscopic code: neurodynamicists extract neural correlates of stimuli and associated behaviors in spatial patterns of oscillatory fields of dendritic activity, which self-organize and evolve on trajectories through high-dimensional brain state space. This multivariate code is expressed in landscapes of chaotic attractors. Unlike other scientific codes, such as DNA and the periodic table, these neural codes have no alphabet or syntax. They are epistemological metaphors that experimentalists need to measure neural activity and engineers need to model brain functions. My aim is to describe the main properties of the macroscopic code and the grand challenge it poses: how do very large patterns of textured synchronized oscillations form in cortex so quickly?

sensory cortices.Its output goes directly to the entorhinal cortex in the limbic system.There all sensory systems converge their patterns, which are integrated into gestalts during passage through the hippocampus.Integration implies that they must use the same basic code, which I identify as the macroscopic code.
In the 1st of three stages in the olfactory system (Fig. 1. left) receptors in the nose use the microscopic code.Chemical information is transmitted by action potentials on axons in parallel that converge to the second stage, the olfactory bulb, and excite mitral (pyramidal) cells.The ~10 8 receptors provide a wide aperture that facilitates capture of faint odorants at small concentrations.In round numbers there are ~10 3 types with ~10 5 of each type.Due to turbulence in the nose a faint chemical excites a different subset of ~10 2 receptors in every sniff.Identification by categorization requires inductive generalization.This is done by associative learning, with pairwise strengthening of synapses between coactive cells on trials with reinforcement, and weakening by habituation on trials without reinforcement.The result is the Hebbian assembly and the mesoscopic code, in which the correlate of the signal is conceptual (a category), not strictly sensory.In the third stage the transmission of the signal is not by topographic mapping but by spatial integral transformation (pp.267-269 in [3]), which like a diffusing lens enhances the macroscopic code 1 .
The same three stages hold in neocortical sensory systems.Those systems differ from three-layered allocortex by the intrusion of neurons that have migrated from the floor of the lateral ventricles into Layer II and formed the extra three layers of neocortex (Fig. 1, right [4]).Specialized neural networks in the intrusive layers preprocess sensory information from the thalamus (microscopic) and transform it to mesoscopic patterns of action potentials.The correlates of the collective firing are exemplified by feature-binding neurons, mirror neurons, etc. [5].The embedding Layers I, V and VI integrate mesoscopic neural activity over great distances, which is feature binding on a grand scale in the transition from the microscopic code to the macroscopic code.
Mutual excitation (Δ, Fig. 1) within the entire assembly (•) amplifies a weak sensory signal, irrespective of which few neurons actually receive the input on each sniff.That is 1 In prior work I based my definitions of 'micro-', 'meso-', 'macro-' on the methods of observation [1]: action potentials; ECoG/LFP (local field potentials); and EEG/MEG/fMRI.Here I base my definitions on behavioral correlates: 'micro-' = information in a material event (sensory or motor); 'meso-' = abstract concepts; 'macro' = experiences (memories).Microevents are described by point processes of single neurons.Mesoevents are point processes of collectives of neurons that are described with neural networks.Macroevents are fields of global interactions in neural populations that are described with the solutions to ordinary differential equations (Ch. 5 in [3]).

Understanding Perception Through Neural 'Codes'
Walter J Freeman, Life Fellow, IEEE.

T
Freeman WJ (2011) Understanding Perception Through Neural 'Codes'.In: Special Issue on "Grand Challenges in Neuroengineering", IEEE Trans Biomed Engin 58 (7): 1884-1890.DOI: TBME-00851-2010.R1 2 inductive generalization.Simulations of population dynamics with nonlinear differential equations [3], [6] show that the negative feedback to the excitatory neurons from inhibitory interneurons (o) generates gamma oscillations (30-80 Hz) (Fig. 2, A), which are regeneratively amplified (B) by small increases in the strengths of Hebbian synapses on association and strongly damped by small decreases on habituation (C). .The networks are embedded in Layers I, V and IV, which extend over the entire surface of both hemispheres.The deep pyramidal cells in Layers V and VI provide the long axons in power-law distributions that are necessary to support the long correlation lengths observed in the multicortical ECoG [7] and scalp EEG [8], [9].Fig. 2. A. Simulation shows the dependence of oscillatory amplitude (B) and frequency (C) on the gain at Hebbian synapse, Kee, [3], [6].(D) Oscillation in the normalized conditional pulse probability density (NCPD) is conditional on the ECoG amplitude and time lag (pp.154-163, [3]).Macroscopic oscillations are not seen in autocorrelations of a microscopic spike train (above), only by using information in the ECoG, which is a spatial sum over a neural population (p.153 [3]).
Dendritic current oscillations modulate firing probabilities of cortical neurons (Section 3.3.2 in [3]).The parsing facilitates mesoscopic Hebbian learning by collimating the firing times of excitatory neurons.The gamma oscillations are macroscopic; they can only be seen by ensemble averaging, because the neurons are time multiplexing.They fire randomly at mean rates (1-10 Hz) well below the beta and gamma ranges, and collectively they rotate the duty cycle.Spike-triggered averaging of local field potentials can often indirectly reveal the oscillation.A better measure (Fig. 2, D) is the pulse probability wave of a single neuron that is calculated from the probability of firing that is conditional on the amplitude and time lag from the ECoG.Oscillations also serve to launder cortical output.They do this when cortical output is transmitted through a divergent-convergent pathway (Fig. 1, left) that by spatial integration enhances endogenous signals that everywhere have the same carrier frequency.The integration attenuates unsynchronized activity by smoothing [2], most notably the stimulus-bound cortical activity that is driven by exogenous sensory input, as distinct from interactively bound activity.

II. THE ELEMENT OF THE MACROSCOPIC CODE
Macroscopic patterns are observed by simultaneously recording the electrocorticogram (ECoG, Fig. 3), which is formed by the instantaneous sum of potentials as the dendritic currents flow across the low extracellular resistance between the neurons.Neuron populations (10 4 to 10 5 ) that are aligned in parallel perpendicular to the cortical surface contribute to the signal seen by each electrode.
In olfaction the bursts are related to the respiratory cycle under limbic control [11].The gating in the other senses is not so clear, perhaps by saccades, tremors, and the cochlear microphonic that is shaped by the muscles of the middle ear.These are manifestations of motor systems that regulate sensory input.In all senses the precise timing of burst onsets is by spontaneous symmetry breaking [12] that recurs in the theta range [13].The firing of neurons in Hebbian assemblies that is triggered by learned stimuli typically lasts up to several hundred milliseconds [5], which is sufficiently long to shape 2 to 3 successive frames.Each burst in a frame is shaped by modulation of the phase, frequency, and amplitude of the carrier wave.The cell assembly selects the temporal carrier frequency at the onset, and the phase spatial pattern is set in the form of a cone (Fig. 3, right).The phase gradient (radians/m) varies inversely with carrier frequency (radians/s), giving an invariant phase velocity (1.89 m/s in the olfactory bulb, 2-4 m/s in neocortex) that is equal to the conduction velocity of intrabulbar axons, not that of the afferent axons from the receptors (0.42 m/s) [2], [14], [15].The location of the conic apex varies unpredictably from each burst to the next, as shown by the dots projected onto the spherical bulbar surface flattened into a circle [14].Most apices lie outside the square array, implying that the gradients extend over the whole bulb.I infer that the correlation distance is limited by phase dispersion with distance from the apex at ±π/4 radians (45°).At this distance the cosine (0.707) gives the half-power radius of spatial coherence, which estimates the correlation length in mm.I adopt the half-power diameter as the soft boundary condition for neocortical gamma bursts [15], beyond which the degree of synchrony may be insufficient to recruit new participant neurons.The modal diameter by this criterion is 15 mm; the 95% inclusion distance is 28 mm (circles centered on AUD, the auditory cortex of the rabbit, Fig. 4), covering large fractions of the cerebral hemisphere that are large enough to include all subareas of each sensory cortical system [16] for binding.Fig. 3.The ECoG of the bulb and cortex has repeated bursts that are initiated by inhalation (sniffing) [11].Each burst has s a conical phase gradient [14].I infer that the apex demarcates the site of burst initiation.The location and sign (•, phase lead; o, phase lag) vary randomly; most apices lie outside the square array.Fig. 4. The macroscopic code is spatial modulation of the amplitude (AM) of the shared frequency in a burst (example at left), here displayed as contour plots.The AM patterns from control and test odor bursts differ during the first day of training in a classic aversive paradigm.Two weeks later the same odorants give AM patterns that still differ from each other [11], but they differ even more from the AM patterns in the first week, thus illustrating the lack of invariance of AM patterns with fixed conditioned stimuli.At right is shown the outlines of 8x8 electrode arrays on the surfaces of the prepyriform (PC), somatic (SOM), auditory (AUD), visual (VIS), and entorhinal (EC) cortices [16].The same macroscopic code holds for all cerebral cortices examined so far [7], [9]- [11], [14]- [17], [18].
Sets of ECoG segments that are centered on bursts display the shared carrier frequency (Fig. 4, left) and the differences in amplitude across the array.Contour plots of AM patterns (Fig. 4, center) show the differences but not categories.To classify AM patterns we measure each burst after analog-to-digital conversion, band pass filtering, and frame normalization (dividing each of the n amplitudes by the mean amplitude of the entire frame) to get n-amplitudes of the synchronized waveform.We construct an nx1 feature vector, which specifies the location of a point in n-space.A stimulus that a subject can perceive on multiple trials gives a cluster of points in n-space.Multiple trials with a different stimulus that a subject can perceive and discriminate give a different cluster.Each cluster has a center of gravity.Categorization of AM patterns is by the Euclidean distance to the nearest center.Unequivocal classification with respect to some measure of intentional behavior is essential for validation of the premise that the AM patterns provide the content transmitted by the macroscopic code.
We have applied this classification procedure to normalized AM patterns in rabbit allocortex [10], [18], neocortex [16], gerbil auditory cortex [17], multiple sensory and limbic cortical signals simultaneously recorded from cat or rabbit [7], and human scalp EEG that was recorded with a standard 64-channel, whole-head array [9].
The classificatory information in macroscopic AM patterns is spatially distributed with uniform density.No channel is more or less important than any other regardless of whether the amplitude, variance, or covariance is high or low [9], [14], [16], [17].Localization in space-time is paramount in microscopic and mesoscopic codes.Macroscopic patterns are not localized; the code is distributed.Classification improves in proportion to the number of electrodes irrespective of location, provided that the array is fixed.
The correlation distances in space of the synchronized carrier waves are many times longer than the mean lengths of intracortical axons (Fig. 4, right).EEG bursts that carry classifiable AM patterns on carrier frequencies in the beta range (12-25 Hz) appear to extend over the entire cortex in each hemisphere in the ECoG [7] and the EEG [8], [9].The distributions of the phase cone diameters of the background ECoG are power-law [15] (Fig. 4, right), indicating that the background ECoG is noise, but the diameters of the phase cones of classifiable AM patterns last up to twice as long as could be expected by chance [13], [15].
Importantly the AM patterns lack invariance with respect to fixed stimuli (Fig. 4, center).They change when the reinforcement is changed, for example, by reinforcing a previously unreinforced stimulus, or when reinforcement is omitted as in the extinction of a conditioned response.These manipulations change the significance or meaning of a stimulus but not the stimulus.AM patterns of learned stimuli also change when new stimuli are learned, and likewise with the passage of time involving the accumulation of new experience [18].Thus the correlate of the AM pattern is not the stimulus; it is the meaning of the stimulus for the subject.On this basis I conjecture that the AM pattern is the unit of the macroscopic code, which corresponds to a thought, percept, word, or fleeting memory.In summary, the microscopic code embodies information; the mesoscopic code embodies abstractions; the macroscopic code embodies thoughts and feelings.

III. PROPERTIES OF THE MACROSCOPIC CODE
Several properties are found for the macroscopic code.
• The frequency of carrier waves is nearly invariant within frames, but it varies unpredictably stepwise between frames [14]- [16], [18].The temporal power spectral density (PSD T ) of long segments (> 1 s) has a power-law distribution in log-log coordinates 1/f α , (Fig. 5, A) [19], [20] with an exponent, 2 < α < 4 [21], indicating that the ECoG in averages over long segments is scale-free [22], [23] in the beta-gamma range.In PSD T of short data segments (~0.1 s) that approximate the durations of bursts, local peaks rise above the 1/ f α trend line, indicating departure from randomness.Multiple peaks above the 1/f α line often coexist at different center frequencies (e. g., Fig. 5, A) [16] but not coextensively, suggesting a soliton-like character of overlapping bursts.
• Owing to the frequency modulation between frames the Hilbert transform is better suited for ECoG/EEG analysis at high temporal resolution than the Fourier transform [24].Each long ECoG segment (~ 6 s) is narrow-band filtered in a set of pass bands in the beta-gamma range (Fig. 5, B).The filterbank/Hilbert transform (FB-HT) decomposes each signal into analytic amplitude (shown for one pass band in Fig. 5, C as log 10 analytic power) and analytic phase or analytic frequency (the time derivative of the analytic phase) in each pass band (Fig. 5, D).
• The high resolution of the FB-HT after pass filtering reveals beats in the filtered signals when the amplitudes of the n-signals approach low values almost simultaneously (Fig. 5, B).This is a form of Rayleigh noise (p.148 [3]), which is produced when white, brown, or black noise is passed through a linear band pass filter.Owing to interference in the frequency mix, the analytic power is briefly near zero (C).The intervals between down beats conform to the Rice distribution [25], [9], [13] for extreme values.Rice proved that the modal interval expressed as frequency in Hz equals 0.641 times the pass bandwidth in Hz [9], [13] for all center frequencies.Empirically the optimal pass band for AM pattern classification is 4-10 Hz, which reveals down beats in the theta range (3)(4)(5)(6)(7).The modal intervals correspond to the intervals between classifiable AM patterns [9], [11], [16].
• When the analytic power approaches zero (Fig. 5, C) between two beats, the analytic phase and frequency are briefly indeterminate (D), giving maximal spatial standard deviation, SD X (t), of the analytic frequency.During a beat the trajectory traced in n-space by the feature vector of the n-signals takes large steps [16], as it transits from one cluster of points to the next.
Freeman WJ (2011) Understanding Perception Through Neural 'Codes'.In: Special Issue on "Grand Challenges in Neuroengineering", IEEE Trans Biomed Engin 58 (7): 1884-1890.DOI: TBME-00851-2010.R1 5 • The analytic phase has a discontinuity (an abrupt change in the analytic phase that is known as phase slip).The prior phase cone vanishes, and a new cone forms with a different apex.Within bursts the SD X (t) stays near a minimum [13].The phase cone is invariant in apical sign, location, and phase gradient in rad/mm [15], [26].The analytic frequency has a low temporal and spatial standard deviation, SD T (t) and SD X (t) [8], [9], [13], [27].The AM patterns of successive digitizing steps form a very tight cluster of feature vectors [16].This configuration of the measures of the parameters of a burst quantitatively define a cinematic frame.
• The frame patterns are finely textured in both phase and amplitude.The textures are not at the scale of single neurons; they are at the scale of cortical columns.The scale of measurement is indicated in Fig. 6, A. The two spikes show the simulated potential field at the cortical surface (the point spread function, so called in analogy to the distribution of intensity in a plane from a point light surface off the plane) of 2 single dendrites or dendritic columns ~0.1 mm in diameter at random locations and at the known depth of the generating layer of pyramidal cells.Such narrow peaks rarely occur in AM patterns.That is because the mutual excitation (positive feedback) among cortical pyramidal cells promotes the co-firing that supports longrange synchronization.
• However, the analytic power in beats intermittently shows dramatic drop-offs in very deep null spikes located in the beats between bursts (Fig. 6, B).The log 10 of power may decrease from peak burst values by a factor of as much as 10 -4 to 10 -6 .In theory [12], [13], [28] the location of the null spike should coincide with the apex of the phase cone (C) of the next burst.Proof that they do so is still lacking, owing to inadequate precision of spatial and temporal measurement of the ECoG signals.New data are needed with higher temporal and spatial resolution given by faster analog-to-digital conversion and by more closely spaced electrodes to avoid undersampling and aliasing.Invariably the temporal correlation among processed signals is close to unity.This reflects the origin of carrier waves from widespread synaptic coupling and not from filter artifacts, volume conduction, or activity at the reference.Evidence is in the fine differences in amplitude and phase modulations of the carrier (Fig. 5, C), which on occasion can approach the point spread function for activation of a single dendritic column (Fig. 6, A, B).

IV. INTERPRETATION OF CORTICAL DYNAMICS
The story these data tell is that in the awake cerebral cortex the limbic system maintains by preafference [29] a macroscopic landscape of chaotic attractors [30], each of which is accessed by a Hebbian assembly.The cortical state of expectancy is characterized by a trajectory in a highdimensional search space, analogous to an aircraft flying over a 2-D terrain.The information in a learned stimulus activates an assembly that directs the cortex into its basin of attraction.The cortical state condenses to a lower dimension as it converges to the selected attractor, which governs the construction of the spatial AM pattern and sustains it in transmission for 3 to 5 cycles of the carrier wave.The reduction in dimension resembles a phase transition with radial spread from a random site of nucleation by either explosion or implosion, as seen in the phase cone with an extreme of either lead or lag at the apex.
The Grand Challenge is to explain how immense populations of neurons can almost simultaneously switch their patterned firings over distances vastly exceeding the range of transmission of all but a few large neurons.The relevant mode of communication within neurons from synapses to trigger zones is by analog currents in dendritic shafts.Between neurons it is by action potentials propagating from trigger zones to synapses.The billions of neurons and trillions of synapses continuously transmit in parallel and time-multiplex signals in randomized pulse trains.By interaction they create collectives with the properties listed above, which differ markedly from the properties of neurons and neural networks.
In brief, the action potential in models of microscopic and mesoscopic networks can be treated as a binary digit, whereas in models of macroscopic population dynamics, the action potential and the brief synaptic potential that it evokes can be treated as infinitesimals in nonlinear differential equations [3], [6].
Cortical neurons create the ECoG by their widespread synaptic actions.The magnitude of power in the ECoG is in part determined by the degree of synchronization of the actions, so the power is a measure of the degree of order that the neurons impose on themselves and each other.In the words of Haken [31] the ECoG serves as an order parameter.When the power approaches zero, I infer that the interactions that impose the order briefly vanish, and that the neurons in that state of disorder are available for capture by a new attractor through a phase transition.
The challenge is met by using the beats and the phase plateaus as guides in the search for signals embedded in what looks like noise [9] and mostly is noise [13].The search is facilitated by use of concepts from many-body physics, which are powerful tools for exploring collective phenomena [12], [28].Among the concepts is criticality [32], a process by which brains hold themselves in a state of readiness to transit between phases of reception and transmission.At criticality the correlation distance can increase to very large values.This concept offers a route to explore the very long distances of synchronization revealed in the scalp EEG [8], [9].
Another useful concept is singularity.Forty years ago I predicted that a singularity would be found in cortical population dynamics.I calculated the characteristic frequencies (roots) of bulbar populations by solving piecewise-linearized ordinary differential equations.Repeated solutions yielded root loci (Ch. 5 in [3]), i. e., changes in roots with increasing burst amplitude.The root loci showed that the olfactory bulb dynamics regeneratively and inexorably approached a limit cycle attractor in the gamma range (Fig. 6.30 on p. 388 in Ch. 6 of [3]), a singularity necessitating a phase transition.Now I propose that the singularity is manifested in the highly localized null spike, the apex of the phase cone, and the center of vortex rotation, expansion, or contraction of the spatial distribution of the band pass filtered ECoG [28], [33].Proving that hypothesis is the present Grand Challenge.

Fig. 1 .
Fig. 1.Left: The three stages of olfaction use respectively microscopic, mesoscopic, and macroscopic codes.(Δ, excitatory neuron) (o, inhibitory neuron) (=•=, member of a Hebbian assembly) [2].Right: Neocortex forms in embryo by intrusion of neurons forming local neural networks in Layers II-IV[4].The networks are embedded in Layers I, V and IV, which extend over the entire surface of both hemispheres.The deep pyramidal cells in Layers V and VI provide the long axons in power-law distributions that are necessary to support the long correlation lengths observed in the multicortical ECoG[7] and scalp EEG[8],[9].

Fig. 5 .
Fig. 5. A. Temporal power spectral density (PSDT).Short segments (0.1 s) show narrow-band peaks of power; long segments (6 s) show "1/f" trends, owing to the wide variation of carrier frequencies.B. Bandpass filtered of 62 superimposed ECoG.C. Log10 analytic power from the Hilbert transform.Note the 10 2 range in signal power among the 64 signals from the 5.6 x 5.6 mm array.The spatial organization of the down spikes at 21.094 s is displayed in Fig. 5, B. D. Analytic frequency (phase difference in radians / digitizing step in s).From [1].

Freeman 6 Fig. 6 .
Fig.6. A. Simulation of the surface potential for two dipoles in cortex.Spacing must be close to avoid aliasing.B. The log10 power in a beat shows a very brief, highly local decrease in power at a point close to the location of the phase cone apex in the following burst (Fig.5, C). C. Phase of the burst in Fig.5.From[1].