How things sound can impact how they feel to us

Whether you’re speaking and listening, singing in an acapella group like in Pitch Perfect, or just enjoying the white noise inherent in being alive, humans have a unique sensory relationship with sound. Unlike sight, it’s not a sense we can choose to turn on or off. Most of us spend our lives awash in auditory input and our brain’s ability to interpret that information significantly impacts our mental model of the world.

With the notable exception of taste and smell — which both play a significant role in the experience of eating — we tend to think of our senses as being discrete inputs, each ruling their own cognitive domain. However, that might not be the case. New research from scientists at the University of East Anglia, and colleagues, reveals that the brain’s different sensory systems are closely connected and overlap in order to build a continuous, if sometimes inaccurate, picture of reality. Specifically, they looked at the relationship between sound and touch, two senses which seem as though they couldn’t be further apart, and found that they work together in interesting ways. The results of their study were posted to the BioRxiv preprint server.

Our common picture of the brain includes distinct regions responsible for specific activities. One might be responsible for processing speech while another handles facial recognition or coordination. When we draw diagrams they have tidy borders, like neural nation states which share data through import and export but don’t otherwise overlap. These visualizations turn out to be oversimplifications which don’t wholly demonstrate the uniquely complex relationships happening inside our minds.

Using functional MRI machines, the team put study participants through a series of experiments in which they were exposed to one type of sensory input, absent any others, and measured what was going on inside the brain. They found that when we hear sounds of familiar objects, the brain regions responsible for our sense of touch activate, even if we’re not actually touching an object.

It’s likely this is a result of previous experience. When you bounce a ball, it has a characteristic sound, and your brain learns to associate those two stimuli. Likewise, when you crumple a paper bag or type on a keyboard. Those satisfying crinkles and key taps cement themselves in your mind as a necessary component of interaction with those objects. When you hear those sounds absent the tactile interaction, you know what they are, and your brain responds by activating your sense of touch.

This sensory overlap doesn’t just occur between sound and touch, though. Researchers also found that if they showed participants a silent video of an object interaction, the auditory regions of the brain were activated. Simply seeing a ball bounce was almost enough to make you hear it, because that’s what your brain expects.

Scientists believe this might be a way for your brain to make predictions about interactions with familiar objects. In doing so, processing those experiences can become more efficient even if there’s an opportunity for them to be less accurate. An unrelated set of experiments involving wine, carried out in 2014, orbited a similar phenomenon when the visual stimuli — whether a wine was red or white — didn’t match the smell and taste stimuli. Wine was presented to ordinary tasters, not wine experts, in black glasses so that they couldn’t see the color. The lighting in the room was then changed to either red, white, or green and participants were asked to describe the taste of the wine. Surprisingly, or perhaps not given what we now know, about 15% to 20% of participants experienced changed taste perception despite the drink being the same every time.

It seems clear that our picture of the world isn’t static. It depends upon a complex combination of sensory stimuli all being interpreted at the same time. This method works pretty well most of the time but if the taste and sight, or the sound and feel of an object are other than we expect, things can get a little weird.