Maker Music Festival: With each drop

With Each Drop is an interactive visual poem, an ode to the intimate relationship people around the world have with water. It is a duet between human and machine, specifically, between a dancer and the text of a poem. All mediated by algorithmic responses with pose recognition generated by machine learning using PoseNet and TensorFlow.

With Each Drop is the collaborative project of a collection of artists and creatives working and experimenting across media, spaces and timezones. The physical and virtual world is our studio. We combined the written word with the movements of a dancer in real time using machine learning to recognize the dancer's postion and pose. Specifically we used the PoseNet pose recognition model with TensorFlow. The piece is written in p5.js for the most part on glitch. It took about a month of cross timezone collaborative work, involving poetry, choreography, coding, composing, video editing and fun.

Side story Hal here, personally I love generative algorithmic music. I also prefer physical objects to screens. I am a part of the Amped Atelier tech couture art practice, we make interactive physical objects, dresses. A couple of years ago we created Conductive Melody a dress as an instrument. The wearer/player/musician plays the dress/instrument by touching eight conductive strips/keys and the notes played in real time are determined/inferred by a machine learning model trained on classical music competitions.
With Each Drop, also hopes to let a dancer also play the music through the use body positioning, and pose classification.

What inspired you to do this?

The project was created for an international tech art collaborative program with the mission to communicate about climate change.

How long did it take to make it?

Once the project had been proposed and accepted, it took about a month and a half to complete.

How long have you been doing things like this?

Some members were new to the tech art scene, one was a season coder, one an established choreographer, one a writer, and one a creative designer.

How much did this cost to do?

Other than time, not much. We used our existing hardware and our existing or free software.

Have you done other things like this?

Using machine learning and pose detection and classification was new to everyone, as was combining dance, written poetry, and creative coding.

What did you wish you knew before you started this?

How hard pose classification was going to be.

Are there plans available to make this? Do you sell this?

It would be wonderful to create a live installation with a magic mirror video wall for real time interaction.
The project lives on line at:
https://witheachdrop.github.io

What’s next?

There is lot that can be improved, from the interactive realtime graphics to fixing pose classification allow a further degree of interactivity.

Resoures?

Want to try it? check out this work in progress demo:
https://witheachdrop.github.io/create/flood.html

We are an international collective of artists and creatives working and experimenting across media, spaces and timezones. The physical and virtual world is our studio.