I’m building a VR technodelic-style game/experience and I need a name!

Music visualizers work by turning sound into a visual experience. Dancing, especially in certain styles, works by turning sound into a kin[esth]etic experience. Here’s a diagram:

Fig 1. music visualization (top arrow) and dancing (bottom arrow)

My idea is to chain these processes, using computer vision and pose estimation, like this:

Fig 2. music visualization via dancing and computer vision

I’m doing this in virtual reality, and I have a working prototype that proves the concept. What should it be called so I can rename my files and launch a Discord?


What Color Are Your Vowels?

One of the most exciting trends in technology is the ability to hook up our sensory and motor systems in splendid new ways. One example is Soundself, a virtual reality “technodelic” that puts you into an audiovisual feedback loop with your own voice. Soundself promised to turn your voice into a psychedelic experience, and I wrote a review explaining my experience with it and why I don’t think it quite lived up to the promise.

In this post I introduce a little demo I made that I call “What Color Are Your Vowels?”, and my hope is that it can illustrate what I think is possible with technodelics. Without further ado, you can try the demo out here. I have only tested it on MacOS (Safari/Chrome/Firefox), Windows 10 (Internet Explorer/Firefox), and iOS (Safari) so no promises it’ll work elsewhere. You start by sustaining three vowels and capturing the background noise level (works better with total silence though) for calibration, and then you’re free to make any other sounds you want and see what color they make.

How it Works

The color space that humans see is three-dimensional because humans have three types of cones in our retinas. If you fix brightness, however, then color space is two dimensional, and you can call it chromaticity space.

Fig 1. Chromaticity space is two dimensional because it fits on an XY plane (Wikipedia)

The dimensionality of vowel space has more caveats, but it’s more-or-less two dimensional, corresponding acoustically to the two frequencies of the first and second formants in the frequency spectrum and anatomically to the two directions of where high-to-low and back-to-front the tongue is.

Fig 2. Vowel space is also two-dimensional (Wikipedia)

What I do with my demo is overlay these two space, mapping each point in vowel space to a point in chromaticity space.

Fig 3. Two 2D spaces, superimposed

It took me ~15 hours and ~200 lines of Javascript to make this demo. This is the kind of thing that I wanted from Soundself. Take my voice, do some linguistically-aware processing, and turn it into compelling visuals that represent it faithfully even in the radically different medium. To do this well you need to have some idea of the parameters that are generated in speech (formants, aspiration, sibilation, rhoticity, etc), some of the parameters that are used in sound visualizers (symmetry, repetition, color, shape, motion, etc) and an artistic flourish in mapping the former set of parameters to the latter set of parameters. There is a lot of potential here to make something truly splendid!

[Footnote] A slightly more technical “How it Works”: I record frequency spectra for your /u/ /a/ and /i/ phonemes, and then take the dot-product of future spectra with those recordings for the R, G, and B levels, respectively. Feel free to dive into the source code, also on my GitHub.

SoundSelf Technodelic Review

After weeks of anticipation in the midst of quarantine-fueled boredom, SoundSelf finally released today. SoundSelf pioneers the self-styled “technodelic” genre of games, and is the first to launch to a wide public audience. Technodelics harness the power of technology to create integrated experiences at the crossroads of psychedelics, meditation, song, dance, and music visualization. They are highly atypical “games”, as they do not have strategy or narrative, but for the most part they can be run on the same gaming platforms, virtual reality in particular. SoundSelf harnesses your microphone, speakers, and display to create a meditative tunnel of light and splendor that echoes your own ahs and oms into a session like you’ve never experienced before.

SoundSelf should receive praise for planting a tall flag in the technodelic turf. I must, say, however, that I was not blown away. I’ve been following technodelics for a year or two now and virtual reality for over a decade, so I’ve been itching to get my hands on those new experiences that I know are possible. What I want is to close the sensorimotor loop of my head, larynx, shoulders, knees, and toes on the one hand and my visual and auditory fields on the other. I want rich feedback between playful fluctuations of voice or step and color, flicker, timbre, beat, and fractals. I want to sing my way through the Mandelbrot Set and dance a virtual sun over a scintillating Dalí horizon. What I want is clearly much more than I could ever have expected SoundSelf to deliver today, but I was hoping for maybe 10% of it.

SoundSelf is essentially a meditation enhancer. Many of the features and constraints (like the smooth-spoken tutorial every time you launch it that you can’t turn off and the 3D perspective that resets whenever you move your head too far) are clearly built in to accommodate that one use-case. There is only one tunnel you can go through. You can’t modulate or change the gain on how your voice affects the visuals. You can barely tell what your voice is doing to the visuals unless you run careful and flow-destroying experiments lasting minutes at a time (I want to hum “ee” instead of “oo” and see something change—I’m not sure what, maybe a triangle turn colors—before I take my next breath). You can’t change the echo interval. You can’t change the music. If what you want is to lie back and be somewhat passively and mindlessly entertained, SoundSelf will work for you, but what I wanted was a vocal technodelic: nimble and pulsating audiovisuals responding to my every formant, susurration, yodel, and yawn.

Do I recommend SoundSelf? I definitely recommend putting technodelics on your radar, and SoundSelf is currently the best way to do that, and there’s no way to know what they’re about without experiencing them yourself. The creators are eager and energetic, so I have faith that they will continue to expand the experience, but keep your expectations tempered, and take it for what it is: a first impression on the debut of an entirely new art form. Throw in a few dollars with a friend and try it out, it’s definitely worth the cost of a theater ticket.