AI pianist gracefully slays the classics, but glitches up on 'unplayable' billion-note tune

Sure, we’d rather watch tuxedoed Tom play Franz Liszt’s "Hungarian Rhapsody No. 2" in Tom & Jerry’s famous “Cat Concerto” episode, but there’s no doubt this AI virtual pianist can tickle some mean ivories in his own right.  

A stunning creation from Canada’s Massive Technologies, the AI is fed audio files, then spits out gorgeous, perfectly timed animation files. The AI was initially intended to be a piano teacher, but those who can... do, and this virtual pianist certainly can, and mesmerizingly so.

Granted, the hands that you see in the videos below aren’t actual, but rather a 3D rendering programmed to accurately caress the keys of any song… well, almost any song. Before we get to the virtual pianist’s blindspots though, let’s celebrate its triumphs.

In the video below, after conquering Tom’s Liszt interpretation, the virtual pianist (let’s call him Beeth Oven) effortlessly plays alongside accomplished pianists performing intricate works by Chopin, as well as the theme from the Detective Conan (aka Case Closed) manga. We also glimpse some scintillating side-by-side action of our Beeth playing next to Joe from Pixar’s Soul. 

But while both animations are hypnotic and beautiful in their own right, one apparently took a lot less time. “The hands in Soul (left) are manually animated frame by frame by multiple animators. Our AI generated a feasible animation (right) in less than 3 seconds from only the raw audio,” reads the graphic. Granted, Joe’s performance is perhaps a bit more emotive.

The video, posted recently by Massive Technologies’ co-founder Fayez Salka, teases the possibility of turning “any audio file into 3D virtual piano concert with one click,” while noting that the AI extracts notes from the raw audio, and generates “the appropriate playing technique, hand, and body motion.”

Until it doesn’t. 

As you can see in another video posted by Salka, Beeth doesn’t fare nearly as well while trying to play supposedly unplayable (at least by two human hands) “black midi” compositions, which can include billions or trillions of notes. Careful though, if the composition made Beeth this glitchy, it may also have an adverse effect on you.

To train the AI, Massive Technologies invited accomplished pianists to play till their fingers bled (or they got tired) under the watchful eyes and ears of “special hardware and sensors,” according to Salka (via VICE). The AI’s work would constantly be given feedback by the pianists and machine learning engineers.

“We would then take that feedback and use it as the curriculum for the AI for our next session with the pianist,” he said. “We repeated that process until the AI results closely matched the human playing technique and style.” 

Massive Technologies is looking to build on their 2017 app that gave the virtual pianist some pre-selected tunes. The company has a new app, currently in Beta now, that allows Beeth to show off his listening/processing skills. Users will be able to just point their phone at any audio source and soon get a vision of Beeth effortlessly tickling the ivories. And Salka still hasn’t given up dreams that the virtual pianist will someday make a great teacher, but for the foreseeable future, it’s more of a watch and learn situation.