How the brain can see sounds
Artists have been fooling the brain for centuries. But research reveals that our senses are up to some tricks of their own, says Roger Highfield
WHEN it comes to understanding the most complex known object in the universe, the wrinkled 3lb lump of tissue that resides in our heads, we often ignore the evidence of our own eyes.
In the past few months, a series of scientific studies has shown that what we see can be profoundly influenced by our other senses, continuing a programme of research that has been unwittingly conducted by artists for thousands of years.
A painter's ability to extract the essential features of an image and discard redundant information is essentially the same function that our visual system evolved to make sense of the world.
Cells in the retina and brain detect a boundary between light and dark as a line. Similarly, artists know which elements are important to convey a good representation to the brain of what they can see. And it is possible that some abstract kinds of art, such as cubism, may have unconsciously exploited the rules and systems that the brain uses to depict form and artificially enhanced the activity with drawings and pictures.
Artists also know how to break rules and confuse the systems to create illusions. Take for example, Enigma, painted in black, blue and white by Isia Leviant. Though static, close inspection reveals that the the solid rings of the picture appear to be moving.
Scientists, too, love to trick the brain because this sheds light on how the cinema of the mind is created. A metronome in which the swinging bar changes from red to green and back again reveals colours moving the opposite way to reality at certain speeds, suggesting that perception of colour and motion are separated by a time delay and that the brain works harder to process information about the motion of an object than colour.
In the past few months, scientists have discovered that there is much more to seeing than meets the eye, revealing still more of the brain's secrets.
In one study on 33 volunteers, subjects were told to say whether a dim, obscured light appeared soon after a sound was made. The researchers found that the light was detected more accurately when it appeared on the same side of the person as the sound: what people hear influences what they see.
"Our results suggest that you will see an object or event more clearly if it makes a sound before you see it," said Prof Steven Hillyard of the University of California, San Diego, who reported the work in the journal Nature.
It has been known for some time that people perform better on a task that involves perception if their attention is drawn to the right place. A few days ago, however, Dr Ladan Shams of the California Institute of Technology, Pasadena, showed that what you see is what you hear: when a single flash is accompanied by several beeps, we wrongly perceive it as several flashes.
Unlike the earlier study, the effect studied by Dr Shams does not involve drawing attention to the right spot, since the sound can be from any location, so it probably taps into different brain circuits, she said.
What these studies emphasise is how our vision profits from signals from other senses. Dr Sham's work shows how one sense can deeply influence another to cause a radical change in perception. And she suspects that this mixing occurs at various levels of processing in the brain, from the areas that interpret raw signals from the senses to those that put processed signals together to create the big picture.
This is supported by using scanners to measure brain activity. In one study published in the journal Science by Dr Emiliano Macaluso of University College London, with Prof Chris Frith and Prof Jon Driver, it was found that a sudden touch on the hand boosts the ability of subject to respond to lights near the same hand.
Critically, this relationship between vision and touch affected activity in brain regions previously thought to be involved only in vision. This mixing of different faculties boosts the bang per buck that the body obtains from the senses and has also been observed in other circumstances. For example, we are better able to understand a person's speech when we can see their lips, and the same mixing of the senses is exploited by a ventriloquist to "project" sound to the visibly moving mouth of a mannequin.
This undermines the traditional view: that the senses work independently, the brain processes information from each sense separately, then puts them together to build up a seamless picture of events around us.
One fascinating issue is how all these findings are linked with synaesthesia, which could be an extreme version of this sensory mixing, when people report mixed up senses in which they can see sounds, or hear colours. Indeed, more abstract concepts can end up being scrambled with the senses. A recent report in Nature by Prof Mike Dixon and colleagues from the University of Waterloo, Ontario, described a woman who believes that five plus two equals yellow.
When the subject, only known as 'C', views numbers, each figure elicits a "photism", so that viewing the number seven generates the colour yellow, 999 is seen as three orange nines and 1000 as a grey one and three white zeroes.
What is striking is that, for the woman, just as seven is yellow, so too is "five plus two". In other words, she sees yellow whenever she accesses the meaning of the number seven, said Prof Dixon. This suggests that her condition is driven by activating the mental concepts of digits rather than seeing the figure itself.
Prof Dixon is now planning a study with brain scanners to find out which brain areas are involved. Perhaps the mechanisms that enable the brain to mix the senses are left permanently "on" or altered in synaesthetes.