While scrolling through product ads, you may notice a pattern: the picture does all the work, and the sound is an afterthought. Yet the moment a viewer sees hands around a product, it is the sound, not the visuals, that tells the brain whether the object is cheap or well-built. This is where vision shows the motion, but audio convinces the brain that the motion is real. Therefore, understanding the neuroscience of crossmodal congruity reveals how pairing visual touch with tactile audio tricks the brain into simulating physical interaction with the product.

Key Takeaways

• Escape the “Muted Interaction” Trap with Consequential Sound: Sounds generated by a product’s normal use (such as a cap shutting closed or the rev of a motor), function as diagnostic cues that the brain uses to infer properties such as power and weight.
• Trigger Perceptual Re-enactment: As prior touch experiences are stored as memory traces, a synchronized use of product sounds can reactivate the brain’s recollection of that touch to fill in the sensory gap left by the screen.
• Frame for a Process Mindset: The effectiveness of a product sound only shifts perceived power and willingness to pay when the ad also encourages viewers to mentally simulate the step-by-step act of using the product.
• Match Sound Design to Product Complexity: This is most effective for products where an underlying attribute is important towards the purchase decision.

The Neural Blueprint: Three Scientific Mechanisms

Crossmodal Modulation of Touch Perception

Stanton and Spence’s (2020) review in Frontiers in Psychology explains the foundational mechanism: auditory cues not only accompany a physical interaction but also recalibrate how that interaction is perceived even when the sound is inaccurate relative to what is being presented. The review demonstrates that amplifying high-frequency components of a touch sound makes a surface feel rougher, while the opposite makes it feel smoother, despite the physical surface never changing. This is clearly demonstrated in the “parchment skin illusion” (Jousmäki & Hari, 1998; Guest et al., 2002), where people rubbing their own hands together while hearing an amplified high-frequency version of that sound reported rougher and drier skin. However it is important to note that:

• The effect is perceptual — physiological arousal data suggests the brain is treating the illusory material change as real.
• Incongruence carries a cost — When auditory and visual information conflict (such as due to mismatched timing or inconsistent acoustic content), it lowers processing fluency and is reported as unpleasant.

Perceptual Re-Enactment and the Digital Haptic Gap

Petit, Velasco, and Spence (2019) address the constraint of the online environment: touch, taste, and smell have never been transmissible online, limiting online marketing. Instead, they propose perceptual re-enactment, where a past touch experience is stored as a memory trace, and a partial cue later on can reactivate part of that stored trace. The review goes on to cite fMRI evidence that images of food activate the gustatory cortex. Additionally, Ho et al. (2013), shows that within a virtual clothing try-on environment, adding a synchronized fabric sound as the user moved led to an increase in willingness to pay for the garment in comparison to when tried in silence.

Consequential Sound and the Process Mindset

Ringler, Sirianni and Christenson (2021) explore why the first two mechanisms do not always translate into sales within the Journal of Retailing. In their four studies, they isolate consequential product sounds: noise generated by a product’s normal operation. From this, it was revealed that loud consequential sounds raised perceived power and willingness to pay, but only when paired with framing that induced a process mindset within System 1 (Escalas & Luce, 2003). However, when participants used an outcome mindset instead, the amplitude effect disappeared, so loud and quiet sounds produced no significant difference in willingness to pay. Therefore, the authors explain this by drawing on cue diagnosticity theory, and that product power can be assessed by imagining the product running, best done through a process mindset.

When Does This Work Best?

• Underlying and unobservable attributes matter: These effects are strongest for products where power, build quality or sturdiness are crucial to the purchase decision and cannot be verified from sight alone.
• Framing has to do the interpretive work: Amplifying a contact sound without also cueing the viewer to imagine using the product risks the sound going unprocessed.
• Congruence is non-negotiable: As seen with Stanton and Spence, mismatches between the sound and visual movement do not only underperform, but actively reduce perceived quality.
• Sensory ambiguity increases the payoff: Auditory cues have more influence when the visual or tactile channel is already ambiguous.

Conclusion: Two Complementary Systems

Moving forward, sound design and process-oriented framing should be considered and used together during ad design. While isolating and amplifying the acoustics of snaps, clicks, or textures of a product interaction makes the digital haptic gap bridgeable, it should also be remembered that the sound only converts into perceived value once the surrounding phrasing and imagery put the viewer into a process mindset that makes the sound worth interpreting. Syncing accurately will allow the sound to do what touch cannot, but framing effectively will further get the brain to listen.

References

Guest, S., Catmur, C., Lloyd, D., & Spence, C. (2002). Audiotactile interactions in roughness perception. Experimental Brain Research, 146(2), 161–171.

Ho, C., Jones, R., King, S., Murray, L., & Spence, C. (2013). Multisensory augmented reality in the context of a retail clothing application. In K. Bronner, R. Hirt, & C. Ringe (Eds.), Audio branding academy yearbook 2012/2013 (pp. 167–174). Nomos.

Jousmäki, V., & Hari, R. (1998). Parchment-skin illusion: Sound-biased touch. Current Biology, 8(6), R190.

Escalas, J. E., & Luce, M. F. (2003). Process versus outcome thought focus and advertising. Journal of Consumer Psychology, 13(3), 246–254.

Petit, O., Velasco, C., & Spence, C. (2019). Digital sensory marketing: Integrating new technologies into multisensory online experience. Journal of Interactive Marketing, 45, 42–61.

Ringler, C., Sirianni, N. J., & Christenson, B. (2021). The power of consequential product sounds. Journal of Retailing, 97(2), 288–300.

Senna, I., Maravita, A., Bolognini, N., & Parise, C. V. (2014). The marble-hand illusion. PLOS ONE, 9(3), Article e91688.

Stanton, T. R., & Spence, C. (2020). The influence of auditory cues on bodily and movement perception. Frontiers in Psychology, 10, Article 3001.

‍