WebRTC & Posenet

Link to Sketch: https://editor.p5js.org/djg509/sketches/Y7aebydYz

To make sound into music, the body traditionally requires an instrument that organizes the effort, and in this sketch we explored an instrumentless social creation of sound. It is bare, raw and minimal. When a keypoint in my two dimensional rendering intersects a keypoint in yours, we start a tone. The tone we create is blunt and simple sine oscillation; the pitch is low if you intersect parts on the left side of the canvas, and high if you intersect on the right. Most intersection happens briefly in the middle. Amplitude rises from bottom of the canvas to the top, but again it’s harder to touch at the bottom of the canvas, so sound doesn’t change much. The implementation is turning a canvas click from p5 documentation(https://p5js.org/reference/#/p5.Oscillator), into a pose intersection on the sketch canvas. What if it took two bodies touching in 2d space to make a mouse click?

First we touch noses for a wonky whooping. Then we see if there’s any change in sensation when we touch a wrist to an ear for the same wonky wooping. One of us notices that to get sufficient distance from the webcam stretches the cord of a headphone to it’s full length, like a leash on a dog. They refuse to unplug the headphones. One of us notices that two bodies in flat two-dimensional keypoints don’t seem to interact. More they overlap like thin shadows. What music does a shadow make when it meets another shadow? Our sense of disconnection is amplified. The distancing between our nose and our eyes and our ears seem different than what I would have imagined, but somehow I take the algorithms word for it. 

A simple sketch in three iterations. A flat tone on the nose. Then a pitch that wobbles and fades as our noses move together. Then a fading wobble when a wrist’s pointillist shadow crosses an elbow’s, getPart(pose,”rightWrist”) meet getPart(partnerPose,”leftElbow”) –displaced ear meets ear. Some control of synthesized sound. Some control of keypoints. Ultimately, a connection between two bodies to amplify the yearning for real human contact, rather than sooth it.

10/5 WebRTC & the Sonified Pose

A pose makes no sound. It’s music is rendered in its silence. Give a pose a sound in this sketch, and it has little natural metaphor. What experience is there for the form to inform? The noises the body makes are certainly distinct from the synthesized sounds of the machine. The interactions between people are in three dimensions, and giving the sketch sound makes the forms seem more like shadows than bodies.

We change the environment. Pop music shines in all the warmest, brightest, most familiar audible spectrums, directly into our ears. The body turns to it like the leaf turns to the sun.

The form then, is to be a shadow in the light. A shadow is more than a single point, so the 4th iteration gathers any collection of points. The 5th iteration, we find the intersection of all the points we choose. Select all the keypoints of the eyes, ears and nose, and you have the rough roundness of a shadow. In the 6th iteration, overlapping shadows obscure the light. When the shadow dancers cover each other, the vox disappears. Together they’re left considering a simpler light.

The attempt was shadow, but what stands out from the sketch is subtler. As if the bodies were some flat quarts —prisms dissecting the ear’s light into the ear’s rainbow. In the digital medium, sound is light anyway...

The form informs the experience.

Pitch, yaw and roll.

Nod your head up and down.

Turn your head left to right.

Roll your head from shoulder to shoulder.

If not the whole body everywhere, perhaps the head. 

Mach1 is a framework for spatial audio. That means that sounds change as the position of your head changes. 

How does the position of your head change? Pitch, yaw and roll.

How does the sound change? The relative volume of each track in the mix goes up or down in each ear, as the pitch, roll or yaw of the head changes.

That’s it. Pitch, roll and yaw in. Dynamic volume changes in each ear, out. How do you get pitch, roll or yaw of the head? The idea is that sensors in your earphones can derive the data. That’s still a couple years from wide adoption. How else can you get pitch, roll and yaw? From a ML prediction of a face in view of a webcam. And it works pretty well. 

Mach1, however, is new. And the machinery for mixing audio tracks and creating the encoding that determines their dynamic output volume changes based on pitch, roll and yaw input... isn’t well documented. 

Ultimately this effort resulted in articulating a clear question to the Mach1 team about under-documented examples. The simplest version of the story, is that the audio files that go into the Mach1 web audio player are to be named T1, T2, T3, T4, B5, B6, B7, B8. No where are the contents of those audio files described.

The output from the Mach1 DAW plugin for mixing spacial audio has a completely different naming convention. And the mapping between the mixed DAW output, and the spatial audio player input, isn’t described anywhere. End of the road, until Dylan at Mach1 responds to my slack message. 

For now, we do something similar, but much less graceful. We progressively layer the unmuting of a 4 part vocal piece,  from low to high, as we roll pitch from -20 to 20 degrees. 

The sound changes as the head moves. Is this even spatial audio? Hard to tell. 

The form informs the experience, but it’s far from obvious. It’s a subtle as the nuance of classical music is.