Convert audio and movement to MIDI so you can control a digital instrument with any sound source including physical instruments or your voice.

I created an application using C++/JUCE that converts incoming audio, either from a musical instrument or microphone, into MIDI Notes. It then sends those notes to your Digital Audio Workstation (DAW) or soft-synth. The application also receives Accelerometer and Gyroscope values wirelessly from an Arduino, and converts those to MIDI CC values that control parameters in your DAW or soft-synth.

You can sing or play a musical instrument, and use it as a MIDI Controller. Because the application is a standalone app, not a plug-in, the DAW sees the app as a MIDI Controller. So you can assign the different sensor values to control whatever parameters you want within the DAW using MIDI Mapping. You can also send the MIDI data out of your computer to control MIDI Hardware.

There is more to the music we hear beyond the notes coming out of a musical instrument. The physicality of instruments impacts the way musicians play them, impacting the music beyond the instrument's sound. By capturing the audio and physicality of instruments, and converting them to data, in this case MIDI, any instrument can be used to create any sound, in any instrument style.

Since the application communicates wirelessly with the Arduino, the sensors can be separated from the associated instrument and musician. They could be mounted on other performers, or perhaps dancers, so that there is interactivity between different performers and the timbral qualities of the digital instrument.

Side story This whole thing came about because I wanted to be able to play my bass guitar and have crazy synth sounds come out for Drum & Bass with live instruments. The rabbit hole kept getting deeper.