I was honored to be asked to help score a film, a romantic comedy, and I used a form of AI music to do it. The film already had high production quality music in place, sourced from a music library. The filmmakers were not satisfied with the music in the movie, and as soon as I viewed the screener, I could see why.
The timing and tone of the music were only intermittently in sync with the picture. That’s common with the use of library music, where the pace and timing of the acting and editing are not what the composer and musicians had in mind when they were recording. It’s especially a problem with physical comedy. You know, timing is the secret — of comedy! And the film definitely is a fun comedy.
Getting down to Pro Basics…
When filmmakers use preexisting music, they’re usually liking some part of the music. something that does fit the show. So, the question is: what part? My instinct here was that it was the drums. The drum line had a groovy live swing beat that just sounded fun and high energy.
I used one of my favorite AI stem extraction programs to isolate the drums. It worked great. We got a swinging virtuoso high energy drum part. And when I adjusted the tempo, tone, and placed it appropriately in the picture, it worked! They loved it. Now it was funny! (Syncing to picture is actually very demanding and exacting work).
Similar technology was used to extract John Lennon’s vocals from a demo recording used to create the Beatles’ last song Now and Then.
The next step in production might be to write out an actual drum score and hire a drummer and studio to record it exactly right to picture. This production had no budget for recording… so… what to do?
In theory we could have used samples and programming to create complete fake drum parts, but few people would say those artificial parts can stand comparison with the real thing. And the filmmakers did care about it sounding good and real. Filmmakers usually do, when it all comes together, and so do audiences, when given the choice.
The production had already paid for the library music, and we now had a legit recording of a live drummer, drawn from the mix by AI. I processed and edited together the best clips from the AI stems, changing tempo, adding accents and bridging material that I generated from samples, et cetera. I now had a hybrid live/sample score — something that happens all the time, but now with a high-tech twist — and had put it to picture.
It worked! The movie is now released with my score.
AI is Still Just AI
However, it’s important to know the AI music isn’t perfect. This was not generative AI (GAI), but it still had those issues of rhythm and timber having strange glitches. In context of dialog, sound effects, etc, it isn’t very noticeable. The drums help drive the action. The tempo changes were done in a modern DAW (digital audio workstation), and didn’t add many artifacts, but if you listen to just the music you can tell it’s got AI.
If you’re wondering, did I try using GAI? Yes I did, and it definitely didn’t work with picture. The results had similar issues to the library music with bad timing to picture, wrong vibe, and AI audio glitches. Games have issues with AI too.
Music made with generative AI frequently wobbles or glitches in pitch, time, and volume, making it pretty distracting, apart from the obvious problem that it really has no way of synchronizing to the action on screen, since the AI doesn’t have the ability to see what it’s scoring to. The simpler stem-extracting software has some of these glitches too, but it’s much easier to fix when it’s just one part. Of course GAI audio is getting better technically, but the soul and personality of a living person are intangible properties. Here’s the topic from an engineering point of view.
Generative AI audio works the same way AI image generation does, with a noise field. The noise is gradually fixed and smoothed out until it sounds like the intended output. You can hear that noise field as a frequently subliminal but constant hiss that never goes away. Decades ago we would interpret this artifact as tape hiss, and nearly all GIA audio has this problem to some degree.
GAI can work with picture if the story needs more ambient, loosely timed music. (That’s a whole other topic, the four levels of media music timing). The secret there is designing your prompt to have the right style, timber, and pitch range. Getting the AI to follow the brief is more luck than skill, but skill helps. I could write another entire article, or series of articles, just on that topic alone.
The filmmakers are friends and I really want their film to succeed. My instinct is the audience will like it on the merits of the film. I can hardly wait to find out.
David Raiklen wrote, directed and scored his first film at age 9. He began studying keyboard and composing at age 5. He attended, then taught at UCLA, USC and CalArts. Among his teachers are John Williams and Mel Powel.
He has worked for Fox, Disney and Sprint. David has received numerous awards for his work, including the 2004 American Music Center Award. Dr. Raiklen has composed music and sound design for theater (Death and the Maiden), dance (Russian Ballet), television (Sing Me a Story), cell phone (Spacey Movie), museums (Museum of Tolerance), concert (Violin Sonata ), and film (Appalachian Trail).
His compositions have been performed at the Hollywood Bowl and the first Disney Hall. David Raiken is also host of a successful radio program, Classical Fan Club.
