Work with Mark: Shadowplay

Idea

Yet another system Mark d'Inverno and I worked on but never finished had the working title 'shadowplay'. We had this idea about an audiovisual installation where people's limbs (or outlines of bodies) would represent grid worlds. Agents would live in these worlds and evolve differently depending on things like limb size, limb movement over time, limb shape and limb position. The agents would make different music/sounds depending on the world they live in. A limb world could be thought of as a musical part in a score. The worlds would sound simultaneous but panned to different speakers to help interaction.

The visitors would see the outline of their bodies projected on a big screen together with the agents represented visually in this picture as tiny dots. Hopefully, people could then hear the agents that got caught or breed inside their own limbs. We hoped to active a very direct feeling of caressing and breeding your own sounding agents.

There were plans for multi-user interaction: if different limbs/outlines touched (e.g. users shaking hands), agents could migrate from one world to another. There they would inject new genes in the population, inflicting the sound, maybe die or take over totally. Though to keep agents within the worlds they were made to bounce off outlines. But one could shake off agents by moving quickly or just leave the area. These 'lost' agents would then starve to death if not adopted by other users.

Tech

The whole thing was written in Processing and SuperCollider. Processing did the video and graphics: getting the DV input stream, doing blob tracking (using the 3rd party library blob detection) and drawing the agents and lines for the limbs. SuperCollider handled rest: the sound synthesis, the genetics, agent state and behaviours, keeping track of the worlds etc. We used a slightly modified version of our A4 agent framework I wrote about in the following post: /f0blog/work-with-mark-bottom-up-approach/.

The two programs communicated via a network (OSC) and would ideally run on different machines.

I had major problems with programming. The math was hairy and all the features were very taxing on the CPU. We never got further than a rough implementation.