Sound design greatly contributes to the feelings of the audience while watching a movie. Valuable information about the emotional aspects of a film is contained in the sound design, so understanding how it works and what it tells you is a key aspect to grasping what your content is all about.
Vionlabs has developed an Emotional Fingerprint API that instead of using sub-par metadata to try to provide users with better content, measures the emotional aspects of various key components, such as sound, color, and pacing. This enables OTT providers to better understand their content, which in turn ensures that they can make better content recommendations.
Since we know what benefits a deeper understanding of the sound of a movie can give to OTT services, let’s examine how sound, just like color, can be used to alter how a cinematic moment is perceived emotionally by its audience.
How Sound Design Triggers Emotion
Looking at the clip from the episode “The Long Night” in the final season of Game of Thrones (2011–2019) listed above, your mind is instantly drawn to what you can see. We observe the whitewalkers climbing the wall and the people waiting in suspense trying to defend it. We see fire turning the environment into a murky brown color. But if you think about it, the story is actually mostly told through sound. This happens to be the darkest Game of Thrones episode, sometimes rendering its visual aspects too dark to make out, thus putting greater emphasis on the sound. The clip features sounds of people calling out to each other to defend their wall, the grunts of the white walkers, and the sizzling of fire. The music has a distinct ticking in it, alerting us that time is of the essence.
And then, timestamped at 00:20 in the clip, we also hear something called a Shepard tone. An effect used quite heavily by filmmaker Christopher Nolan, this sound creates an illusion, a scale that seems to be rising forever without ending. This effect is created by superimposing different tones separated by octaves on top each other and tinkering with the volume of the frequencies. The result is what one might call the sound of suspense; and in this particular Game of Thrones moment it is key to the emotional buildup we feel as an audience.
The truth behind great sound design is that it helps you truly immerse yourself into the story, so much so that you might not even consider that it is there. Whether it be the tumultuous thunder of weaponry in Saving Private Ryan (Spielberg, 1998), the sound of a lightsaber in Star Wars, or the theme song accompanying Indiana Jones as he goes on his adventures, the sound in a film helps create realism, adds credibility to the story and above all, sucks you into the emotional state of the moment.
Sound Throughout Film History — The Short Version
Throughout film history, sound has always accompanied our motion pictures. Even during the art form’s first trembling steps, and what would follow as the silent era, films were never fully silent. While some of the leading industry innovators like Edison had an interest in sound, the technology wasn’t advanced enough to entertain a larger audience. So in the silent era something else emerged to keep viewers engaged — music. Often played by a live musician hired by the cinema, it was the music that made all the difference. It was there to set the tempo, it created the mood, and told the audience what they should feel at any given moment.
In Japan, where filmmakers took a different approach from its Western counterpart, they even went so far as to hire benshi — men who would stand alongside the screen and narrate what was happening in the film. The benshi gained a lot of popularity among crowds and would make sound effects, take care of all the dialogue, and be the true performer of the film.
When The Jazz Singer (Crosland, 1927) hit theaters in 1927, it surprised audiences to hear the actor Al Jolson’s chatty adlibs in between his singing. They instantly wanted more; and while most features included a mix of prerecorded audio and the various story-telling techniques of the silent era, just two years later in 1929, 75% percent of movies had turned into talkies.n
Since then, storytellers and filmmakers have been doing their utmost to challenge how we hear and feel through sound in movies. In Francis Ford Coppola’s Apocalypse Now (Coppola, 1979) for example, sound designer Walter Murch forced both theaters and Hollywood to adopt a six-speaker surround system by changing the way sound was edited and mixed. Murch created sound that was able to travel around the movie theater, allowing the audience to experience the film through the traumatized war-heavy mind of Captain Willard. It is clear that sound has now become a major component of the filmmaking process.
Sound Design and its Components
The sound design of any movie can be broken down into several different parts. Filmmakers are able to manipulate each sound component to contribute to the emotional feelings created in that moment.
Usually, upon filming, the only sound recorded would be the dialogue. Other key components, such as music, weather, the sound of objects, or a character’s interaction with them is all added later. The reason why filmmakers do this is because in order for the audience to fully grasp the emotional moment of a scene, the sound is hugely important. Being able to manipulate the sound after filming is therefore major in creating what you would like to convey. Dialogue, sound effects, music, and mixing all come together to tell you the story.
In a film, there is diegetic sound — which is sound heard by the characters inside the filmic universe. Examples of this is dialogue, sound effects, or music played on the radio, like in the infamous scene in Reservoir Dogs (Tarantino, 1992) where a cheerful song accompanies Mr. Blonde as he performs an act of violence.
There is also non-diegetic sound — which is sound not heard by the characters — often added to make sure the audience feels a certain way. The biggest example of this is music, like the orchestral track featured in Steven Spielberg’s Jaws (Spielberg, 1975), a tune that quickly ensures that suspense and fear are added to the shark’s appearance on screen.
The Emotional Orchestrator of Film — Music
Ever since the beginning of cinematic history, music has been a major component in how sound design triggers emotions. Music can be considered the emotional orchestrator of film. It suggests to the viewer what they should be feeling. For example the piano tune created by the AI in Her (Jonze, 2013) expresses the bittersweet technological love felt between the characters, whilst the intense violins in the Psycho (Hitchcock, 1960) shower scene convey precisely how horrifying this encounter is to the audience.
By attaching a signature melody to the story, filmmakers are able to use our expectations to quickly alert us as to what we should be feeling. Take a look at Darth Vader for instance. The menacing imperial march sets the tone. After hearing it a few times throughout the story, even someone not looking at the screen would be able to tell when the Imperial Empire is present and strong. Without it, Darth Vader would just be a man breathing heavily in a black suit.
The Deep Emotional Cues of Sound Design
But we also live in a world where nonmusical sounds are connected to deep emotional cues. The sounds of cars honking, birds chirping, or waves crashing against a cliff all induce feelings.
Since most of what you hear on screen is actually added in post-production, it gives the filmmakers the freedom to use sound effects to tell the story emotionally. In Gravity (Cuarón, 2013) the sound designers really explored how being inside a vacuum-like environment would affect the sound. In Eraserhead (Lynch, 1977) dreary, abstract sounds not reflected on screen were used to create a soundscape, a mood, and an atmosphere. In Brokeback Mountain (Lee, 2005), different types of wind were used to signal the isolated characters’ shifting emotional states.
Since sound effects provide a sense of realism to what we see on screen, each sound has to be carefully considered. Footsteps will sound different depending on whether the scene is happy or scary, if the character is stomping or sneaking. Foley artists — professionals who recreate sounds for moving pictures — therefore work hard to introduce different elements to enhance a sound in order to make it believable and just right for the story at hand.
And to go even further, sound designers also frequently use something called “sonic metaphors”, where a different sound is added to the mix to create emotional depth. In Mad Max: Fury Road (Miller, 2015), as the War Rig roars through the desert, whale sounds were added to the mix to reinforce the feeling of this vehicle as a metaphor of a heavy, slow animal. As the truck takes a hit, the whale groans. It is not noticeable unless you know it’s there, but it adds emotional depth to the sound of the vehicle. This is a powerful tool that keeps the audience engaged.
How To Become Cutting-Edge by Using Vionlabs Emotional Fingerprint API To Measure Sound
So in times when more streaming services are launched than ever, how can understanding sound be utilized to help OTT providers stand out, become cutting-edge, and reduce churn?
The problem with today’s content data is that it is not detailed enough. It simply fails to detect the nuances necessary in order to give good content recommendations. While it makes sense that the metadata for I, Tonya (Gillespie, 2017), a black comedy about figure skater Tonya Harding and the event that ended her career, and Blades of Glory (Gordon, 2007), the sports buddy comedy where Will Ferrell and Jon Heder decide to become an all-male figure skating duo, would be similar, the two films might not attract the same audiences. But, when using metadata, they would most likely be linked to each other.
Vionlabs’ Emotional Fingerprint API goes further than this. It measures and examines thousands of factors in a film including colors, pace, audio, object recognition, and much more. These factors are distilled by AI into a fingerprint that encodes the emotional structure of the content. Such fingerprints will allow the industry to make better informed decisions, and enable the consumers to come into contact with content they would actually want to consume.
Originally published at: https://medium.com/vionlabs-tech-blog/how-sound-design-triggers-emotion-bf052d3da2a9
Read our other blog posts:
https://www.vionlabs.com/post/how-movies-use-color-to-create-emotion
