Audio-visual interactions: How images affect our perception of music
Sound and visuals accompany each other in many contexts - from films, to apps, to video games - and studies have often explored how music affects our perception of visual information. For example, we know that film soundtracks can add meaning to characters' actions and affect which items from a scene are remembered best. However, few papers have researched the inverse relationship - whether visual information influences our perception of music. Considerations such as this are important to artists and commercial work alike in the pairing of sound and image.
In the early 1980's, MTV popularised a new genre of art: the music video. Some researchers claimed that these accompanying visuals would decrease the appeal of the music by distracting or limiting the listener's imagination. Others believed that the videos would enhance a song's appeal by adding deeper meaning to the music. Whilst the debate continues, research suggests that music videos can prevent "wear out" - becoming tired of a song through repeated exposure.
More recently, researchers Marilyn Boltz, Brittany Ebendorf, and Benjamin Field set out to better understand whether moving image affect more specific aspects of our perception of music. Their results suggest that it does.
The experimenters used two types of visuals to accompany the music: videos and static-image montages. Generally, the mere presence of visuals changed how the music was perceived. Melodies accompanied by visuals were perceived to be faster, louder, and more active and rhythmic than the same melodies heard by themselves.
Images also seemed to intensify the emotional impact of a tune. Melodies that had been judged individually as neutral seemed to take on positive or negative characteristics, depending on the characteristics of the accompanying visuals. The perceived acoustic qualities of a tune (its speed, loudness, and rhythms) also changed depending on whether it was accompanied by positive or negative visuals.
Finally, visual changes distorted melody recognition, depending on whether the sound was accompanied by images with positive or negative characteristics. Negative displays decreased melody recognition, whilst positive displays increased it.
Music for media is traditionally an afterthought. Millward Brown, for example, estimate that brands spend 84% of time and money on visual branding. This present study, along with many others, highlights the importance of considering audio-visual media as interactive. Content creators need to not only consider how music might affect their visual design, but also how these visuals will in turn alter perceptions of their music.