Visual Processing and Action

Successive Fixations Focus Our Attention in the Visual Field

Attention Selects Objects for Further Visual Examination

Activity in the Parietal Lobe Correlates with Attention Paid to Objects

The Visual Scene Remains Stable Despite Continual Shifts in the Retinal Image

The Parietal Cortex Provides Visual Information to the Motor Syste

VISION REQUIRES EYE MOVEMENTS. Small eye movements are essential for maintaining the contrast of objects that we are examining. Without these movements the perception of an object rapidly fades to a field of gray, a phenomenon correlated with the decreased response of neurons in area V1 (see Chapter 25). Large eye movements direct the fovea from one object to another. These movements or saccades bring the high resolution of the fovea to bear on different regions of the visual field, exploiting the high density of photoreceptors in the central fovea. Without saccades this high-resolution processing could be achieved only by moving the head or body.

The preceding chapters have described how visual images are constructed, beginning with the processing of intensity and contrast, then the integration of visual primitives, and finally the high-level processing that leads to the recognition of objects. But the visual system involves more than just object recognition. It must also support the brain’s goal of assigning significance to objects in order to develop strategies for interacting with the environment. Thus the brain must be able to select some objects for greater examination while ignoring others.

In this chapter we consider how saccades support that goal. We first consider the essential benefits that saccades provide, shifting attention in the visual field and assisting with the preparation to grasp objects. We then consider the brain mechanisms that solve a major problem created by saccades—the fact that the retinal image is abruptly displaced with every saccade.

In shifting our attention from how the brain constructs a visual scene to how it uses visual information to plan actions, we now concentrate on the region of the brain referred to as the dorsal visual pathway (Figure 29–1). This pathway extends from V1 to the regions in parietal cortex that continue the intermediate level of visual processing, such as the middle temporal area, and then to other regions of parietal and frontal cortex. The regions particularly relevant to this chapter are in the parietal cortex, such as the lateral intraparietal area, but include also the frontal eye field region of the frontal cortex.

Figure 29-1 Pathways involved in visual processing for action. The dorsal visual pathway (blue) extends to the posterior parietal cortex and then to the frontal cortex. The ventral visual pathway (pink) is considered in Chapter 27. (AIP, anterior intraparietal cortex; FEF, frontal eye field; IT, inferior temporal cortex; LIP, lateral intraparietal cortex; MIP, medial intraparietal cortex; MST, medial superior temporal cortex; MT, middle temporal cortex; PF, pre-frontal cortex; PMd, PMv, dorsal and ventral premotor cortices; TEO, occipitotemporal cortex; VIP, ventral intraparietal cortex; V1–V4, areas of visual cortex.)

Successive Fixations Focus Our Attention in the Visual Field

A saccade usually lasts less than 40 ms and redirects the center of sight in the visual field. Saccades occur several times per second, and each intervening period of fixation lasts several hundred milliseconds. The Russian psychologist Alfred Yarbus was the first to show that the pattern of saccades made by a human looking at a picture reflected the cognitive purpose of vision. He found that saccades were not directed equally to all parts of a scene. Areas of apparent interest were fixated most frequently, whereas background objects were ignored. For example, the faces of people were fixated repeatedly (Figure 29–2).

Figure 29-2 Eye movements during vision. A subject viewed this painting (An Unexpected Visitor by Ilya Repin) for several minutes, making saccades to selected fixation points—primarily faces—that presumably were of most interest. Lines indicate saccades, and spots indicate points at which the eyes fixated. (Reproduced, with permission, from Yarbus 1967.)

The image on the fovea shifts with each saccade, yet we perceive a stable visual world. How does that come about? One possibility is that the brain creates a representation of the entire visual scene from a series of visual fixations across the scene and that what we see is this summed representation of the visual world. If that were so, we should have detailed knowledge of the entire visual scene at any given instant.

A series of experiments on change blindness showed that this is not the case. These experiments involved changing a picture during the brief time when the viewer made a saccade from one part of the scene to another. If there were a relatively complete internal representation of the scene before the saccade, then any substantial change made during the saccade should have been recognized. But even a large change frequently went unrecognized. This change blindness occurred even when there were no actual eye movements, as when two pictures were shown in succession with only a brief blank between them to simulate the effect of an eye movement (Figure 29–3).

Figure 29-3 Change blindness. In this example one picture is presented followed by a blank for 80 ms, followed by the second picture, another blank, and a repeat of the cycle. The subject is asked to report what changed in the scene. There is a substantial and, once perceived, obvious change between the two pictures. It takes multiple repetitions for most observers to detect the difference. (Reproduced, with permission, from Ronald Rensink.)

The results of the change-blindness experiments are inconsistent with the hypothesis that we are continually updating a complete representation of the visual field from second to second. Instead we seem to pay attention to only certain fragments of the scene. This selective visual attention relies on the saccades that bring the images of desired parts of the visual field onto the fovea.

Attention Selects Objects for Further Visual Examination

In the 19th century William James described attention as “the taking possession by the mind in clear and vivid form, of one out of what seem several simultaneously possible objects or trains of thought. It implies withdrawal from some things in order to deal effectively with others.” James went on to describe two kinds of attention: “It is either passive, reflex, nonvoluntary, effortless or active and voluntary. In passive immediate sensorial attention the stimulus is a sense-impression, either very intense, voluminous, or sudden … big things, bright things, moving things … blood.”

More recently these two kinds of attention have been termed involuntary (exogenous) and voluntary (endogenous) attention, or bottom-up and top-down attention. Your attention to this page as you read it is an example of voluntary attention. If a bright light suddenly flashed, your attention would probably be pulled away involuntarily from the page.

Voluntary attention is closely linked to saccades because the fovea has a much denser array of cones than the peripheral retina (see Figure 26–1), and this permits a finer-grain analysis of objects than is possible with peripheral vision.

Attention, both voluntary and involuntary, has several measurable effects on human visual performance: It shortens reaction time and makes perception more sensitive. This increased sensitivity includes the abilities to detect objects at a lower contrast and ignore distracters close to an object. The abrupt appearance of a behaviorally irrelevant cue such as a light flash reduces the reaction time to a test stimulus presented 300 ms later in the same place, but increases reaction time when the test stimulus appears at a different place. The light flash involuntarily draws attention to itself, and attention to that location is maintained for a brief period, thus accelerating the visual response to the later test stimulus at the location. Similarly, if a subject plans a saccade to a particular part of the visual field, the contrast threshold at which any object there can be seen is lowered by 50%. The saccade, under voluntary control, draws attention to its goal.

Activity in the Parietal Lobe Correlates with Attention Paid to Objects

Clinical studies have long implicated the parietal lobe in the process of visual attention. Patients with lesions of the right parietal lobe have normal visual fields when their visual perception is studied with a single stimulus in an uncomplicated visual world. However, when presented with a more complicated world, with objects in the right (ipsilateral) and left (contralateral) visual hemifields, they tend to report more of what lies in the right visual hemifield.

This deficit, known as neglect syndrome (see Chapter 17), arises because attention is focused on the visual hemifield ipsilateral to the lesion. Even when patients are presented with only two stimuli, one in each field, they report seeing only the stimulus in the ipsilateral hemifield. They do not have the ability to focus attention in the hemifield contralateral to the lesion, and as a result they may not see everything in that hemifield, even though the sensory pathway from the eye to the striate and prestriate cortex is intact.

This neglect of the contralateral visual hemifield extends to the neglect of the contralateral half of individual objects. Patients with right parietal deficits often have difficulty reproducing drawings. When asked to draw a clock, for example, they may force all of the numbers into the right side of the face (see Figure 17–11), or when asked to draw a candlestick they may draw only its right side (Figure 29–4).

Figure 29-4 Drawing of a candlestick by a patient with a right parietal lesion. The patient neglects the left side of the candlestick, drawing only its right half. (Reproduced, with permission, from Peter Halligan.)

The process of attentional selection is evident at the level of single parietal neurons in the monkey. The responses of neurons in the lateral intraparietal area to a visual stimulus depend not only on the physical properties of the stimulus but also on how the monkey behaves toward it. When a monkey fixates a spot, a stimulus in the neuron’s receptive field evokes a moderate response. When the animal must attend to the same stimulus, the stimulus evokes a greater response, often by a factor of two. Conditions that evoke both involuntary and voluntary attention—the abrupt onset of a visual stimulus in the receptive field or the planning of a saccade to the receptive field of the neuron—evoke still greater responses (Box 29–1).

Neurons in the lateral intraparietal area collectively represent the entire visual hemifield, but the neurons active at any one moment represent only the important or salient objects in the hemifield. That is, a few salient objects—such as the goal of an eye movement or a recent flash—evoke responses in a subset of neurons, and the activity of these neurons is greater than the background activity of the entire population of cells. Both the attention mechanisms and saccades are directed to the peak of the map.

The absolute value of the response evoked by a salient stimulus does not determine whether the stimulus is the most likely saccade target or most highly attended stimulus. When a monkey plans a saccade to a stimulus in the visual field, attention is on the goal of the saccade, and the activity evoked by the saccade plan lies at the peak of the salience map. However, if a bright light appears elsewhere in the visual field, attention is involuntarily drawn to the bright light, which evokes more neuronal activity than does the saccade plan. Thus the locus of attention can be ascertained only by examining the entire salience map and choosing its peak; it cannot be identified by monitoring activity at one point alone.

The Visual Scene Remains Stable Despite Continual Shifts in the Retinal Image

Saccades create a major challenge for visual processing. Successive saccades produce a series of discrete images, each centered where the eye is looking (Figure 29–8). Although this result might be expected to resemble a home movie with the camera moving about in a jumpy fashion, it does not. How visual scenes remain stable despite repeated shifts in focus has been a source of speculation since the 1600s.

Although the basis of this stability is not understood, changes in perception at the time of a saccade offer clues. At the time of a saccade, objects in the perceived scene do not have exactly the same spatial arrangement as in the visual field. The perceived scene appears spatially compressed such that stimuli presented just before the saccade appear closer to the presaccade point of fixation, whereas stimuli presented after the saccade appear to be closer to the saccade’s target (Figure 29–9).

Figure 29-9 Compression of visual space at the time of a saccade. The perceived location of a stimulus presented just before or after a saccade is shifted toward the target of the saccade. (Reproduced, with permission, from Honda 1991.)

This spatial compression is usually no larger than half the size of the saccade and occurs only when there is a larger visual scene. It is not due to stimuli falling on different parts of the retina because of the saccade, for stimuli presented before the saccade also appear compressed. These considerations indicate that some extravisual information is involved in the processing of saccade commands.

What neuronal mechanism might underlie this apparent shift in images at the time of saccades? Neurons in the parietal cortex alter their activity preceding saccades in ways that seem remarkably related to the perceptual phenomenon. When a monkey is about to make a saccade, a neuron becomes less sensitive to the stimulus already present in its receptive field and begins to respond to the stimulus that will be within the receptive field after the saccade (Figure 29–10).