Tech News

Is it a horror movie or a rom-com? AI can predict primarily based solely on music

Is it a horror film or a rom-com? AI can predict based solely on music
Fig 1. The Rating Stamper pipeline. A movie is partitioned into non-overlapping five-second segments. For each section, Dejavu will predict if a monitor within the movie’s soundtrack is enjoying. Cues, or situations of a track’s use in a movie, are constructed by combining window predictions. On this instance, the “Cantina Band” cue lasts for 15 seconds as a result of it was predicted by Dejavu in two close by home windows. Credit score: DOI: 10.1371/journal.pone.0249957

Music is an indispensable ingredient in movie: it establishes environment and temper, drives the viewer’s emotional reactions, and considerably influences the viewers’s interpretation of the story.

In a latest paper revealed in PLOS ONE, a analysis group on the USC Viterbi Faculty of Engineering, led by Professor Shrikanth Narayanan, sought to objectively study the impact of music on cinematic genres. Their research aimed to find out if AI-based expertise may predict the style of a movie primarily based on the soundtrack alone.

“By higher understanding how music impacts the viewer’s notion of a movie, we acquire insights into how movie creators can attain their viewers in a extra compelling method,” mentioned Narayanan, College Professor and Niki and Max Nikias Chair in Engineering, professor {of electrical} and laptop engineering and laptop science and the director of USC Viterbi’s Sign Evaluation and Interpretation Laboratory (SAIL).

The notion that completely different movie genres are extra seemingly to make use of sure musical parts of their soundtrack is fairly intuitive: a lighthearted romance may embrace wealthy string passages and luxurious, lyrical melodies, whereas a horror movie may as a substitute function unsettling, piercing frequencies and eerily discordant notes.

However whereas previous work qualitatively signifies that completely different movie genres have their very own units of musical conventions—conventions that make that romance movie sound completely different from that horror film—Narayanan and group got down to discover quantitative proof that parts of a movie’s soundtrack might be used to characterize the movie’s style.

Narayanan and group’s research was the primary to use deep studying fashions to the music utilized in a movie to see if a pc may predict the style of a movie primarily based on the soundtrack alone. They discovered that these fashions have been in a position to precisely classify a movie’s style utilizing machine studying, supporting the notion that musical options could be highly effective indicators in how we understand completely different movies.

In line with Timothy Greer, Ph.D. pupil at USC Viterbi within the division of laptop science who labored with Narayanan on the research, their work may have precious functions for media corporations and creators in understanding how music can improve different types of media. It may give manufacturing corporations and music supervisors a greater understanding of learn how to create and place music in tv, motion pictures, commercials, and documentaries with a view to elicit sure feelings in viewers.

Along with Narayanan and Greer, the analysis group for the research included Dillon Knox, a Ph.D. pupil within the division {of electrical} and laptop engineering, and Benjamin Ma, who graduated from USC in 2021 with a B.S. in laptop science, a grasp’s in laptop science, and a minor in music manufacturing. (Ma was additionally named one of many two 2021 USC Schwarzman Students.) The group labored inside the Heart for Computational Media Intelligence, a analysis group in SAIL.

Predicting style from soundtrack

Of their research, the group examined a dataset of 110 standard movies launched between 2014 and 2019. They used style classification listed on the Web Film Database (IMDb), to label every movie as motion, comedy, drama, horror, romance, or science-fiction, with lots of the movies spanning a couple of of those genres.

Subsequent, they utilized a deep studying community that extracted the auditory info, like timbre, concord, melody, rhythm, and tone from the music and rating of every movie. This community used machine studying to investigate these musical options and proved able to precisely classifying the style of every movie primarily based on these options alone.

The group additionally interpreted these fashions to find out which musical options have been most indicative of variations between genres. The fashions did not give specifics as to which sorts of notes or devices have been related to every style, however they have been in a position to set up that tonal and timbral options have been most vital in predicting the movie’s style.

“Laying this groundwork is de facto thrilling as a result of we are able to now be extra exact within the sorts of questions that we need to ask about how music is utilized in movie,” mentioned Knox. “The general movie expertise may be very difficult and with the ability to computationally analyze its impression and the alternatives and tendencies that go into its development may be very thrilling.”

Future instructions

Narayanan and his group examined the auditory info from every movie utilizing a expertise generally known as audio fingerprinting, the identical expertise that permits providers like Shazam to determine songs from a database by listening to recordings, even when there are sound results or different background noise current. This expertise allowed them to have a look at the place the musical cues occur in a movie and for a way lengthy.

“Utilizing audio fingerprinting to take heed to all the audio from the movie allowed us to beat a limitation of earlier movie music research, which normally simply regarded on the movie’s whole soundtrack album with out figuring out if or when songs from the album seem within the movie,” mentioned Ma. Sooner or later, the group is all in favour of making the most of this functionality to review how music is utilized in particular moments in a movie and the way musical cues dictate how the narrative of the movie evolves over its course.

“With the ever-increasing entry to each movie and music, it has by no means been extra essential to quantitatively research how this media impacts us,” Greer mentioned. “Understanding how music works along with different types of media will help us devise higher viewing experiences and make artwork that is shifting and impactful.”

AI instrument characterizes a track’s style and supplies insights concerning notion music

Extra info:
Benjamin Ma et al, A computational lens into how music characterizes style in movie, PLOS ONE (2021). DOI: 10.1371/journal.pone.0249957

Offered by
College of Southern California

Is it a horror movie or a rom-com? AI can predict primarily based solely on music (2021, September 15)
retrieved 15 September 2021

This doc is topic to copyright. Aside from any honest dealing for the aim of personal research or analysis, no
half could also be reproduced with out the written permission. The content material is offered for info functions solely.

Source link