Paint with Music!

[jesseengel]

In collaboration with Google Arts and Culture, we've released a new experience that let's you draw and have your brush strokes converted to music with the javascript DDSP models from magenta.js.

g.co/paintwithmusic

Enjoy!

[realfolkcode]

Nice! If only DDSP could have real-time support... This music drawing app would be much more incredible.
Personally, I dream about playing fancy instruments such as violin and saxophone without actually having them physically. Paint with Music reminded me of my pet project https://github.com/realfolkcode/theremin (currently abandoned it), which allows user to perform music with their hands and webcam. It is really similar to what you made in terms of concept. Hands are tracked with the help of Google's Mediapipe Hands solution. Then, based on the hands coordinates, the sound is produced. With one hand, you can control the pitch (the same way as in Paint with Music, however, my scale is linear instead of logarithmic currently). With the other hand, you can control the amplitude of the signal.
Initially, it was aimed to simulate musical instrument called theremin. But now, it encompasses much broader scope. The idea Is to allow users to basically build their own virtual musical instrument by customizing the timbre and musical scale. I was trying to incorporate Antoine Caillon's realtime DDSP implenetation in my ddsp-support branch. However, I am definitely doing something wrong and the result is not so great. Lately, I've been further toying with Mediapipe and implemented my own gesture recognition system (the killer feature of which is recording your own gestures instead of using predefined ones). I hope someday I can return to my project when you further progress with ddsp and add realtime support! :)