A live interactive projection mapping performance

ekua motion graphic

EKUA: To Get To The Other Side

EKUA: To Get To The Other Side is an interactive music experience that bridges live performance, projection art, and customized audience contribution.

𖤐 Role

UI Design
Visual Identity
Motion Design

✐ Tools

Figma
Illustrator
TouchDesigner

⏱︎ Duration

12 weeks

𖠋 Team

1 Designer
3 Developers

ekua singing on live performance date

Background

An interdisciplinary collaboration between design, technology, and performance art

Our team paired with EKUA, a local Rochester, NY musician, to co-create an interactive music experience in support of her latest project, To Get To The Other Side.

Following several collaborative meetings, we decided to pursue a mixed-reality live performance that utilized projections to map and animate audience customized, dynamic patterns onto the artist’s body and the surrounding environment in sync with the music.

Client Background

Who is EKUA?

Natalia 'EKUA' France is a Ghanaian-American artist whose work exists at the intersection of fashion, visual art, beauty, dance, and music. EKUA’s music blends Afrobeat, R&B, soul, and pop; she is inspired by Bob Marley, Lauryn Hill, Tems, Rihanna and her practice is influenced by her cultural heritage, spirituality, and contemporary expression.

Research

Developing the Experience

Before defining core components of our experience, our team aimed to create an experience centered around rich sensory immersion while integrating EKUA's interest in fashion and dance. Below are two major experiential inspirations.

Franck Sorbier F/W 2013 Dress Projections

Franck Sorbier’s Fall/Winter 2013 collection uses projection mapping to animate a white gown creating a dynamic merging of fashion and technology.

Features

Dynamic Dress Animation

Variable Environments

Quick Transitions

Customizable

Takeaways

Projection mapping amplifies both the model and the environment.

Projection creates opportunity for experimental motion.

teamLab's Sketch Ocean

Sketch Ocean was an interactive installation that had participants color their own fish illustrations, which were then digitally brought to life, and released into a virtual aquarium environment.

Features

Personalization

Real-life Tactile Input

Real-time Animation

Community Interaction

Takeaways

Users are able to contribute personal and meaningful inputs.

Users can interact with other users' inputs

teamLabs interactive exhibit

Goals

Center the music

Ensure that interactive experience never overshadowed the music or the performance itself.

Connect audience and artist

Spark curiosity and excitement for EKUA’s music and new project.

Create meaningful interaction

Every aspect of the interaction should be enjoyable, engaging, and fun.

Project Management

Defining Project Components

As the team lead for this project, I defined the key tasks required to bring our interactive performance concept to life. The following outlines the main tasks our team focused on throughout the project.

Interactive coloring app

Develop a web-based application.

Static and moving visual assets

Design branding and animated performance assets.

TouchDesigner integration

Create a projection mapping pipeline that live animates customized input.

Event logistics

Coordinate equipment needs, production schedule, and user experience.

User Flow

Live Event Planning

On the day of the performance, our team would need to live transfer audience submitted colored patterns into TouchDesigner. I created a foundation event flow that allowed us to keep track of necessary performance steps.

ekua event user flow

Design

App Interface Design

The primary goal for the coloring app was to create a simple flow that encouraged users to be able to create as many patterns as possible with minimal effort on the performance day.

Moodboard

EKUA wanted to draw from traditional Ghanaian textiles and West African spirituality. I created a mood board with EKUA to establish visual themes and decided on visuals centered around vibrant dynamic patterning.

ekua interactive coloring application

Simple Patterns

We conducted user testing with our preliminary app featuring a variety of patterns and found that users applying color to more complex patterns took longer contemplating their choices, slowing their output. The simplest patterns were most impactful and allowed for more creative freedom.

pattern 1

pattern 2

pattern 3

pattern 4

pattern 5

pattern 6

Live Body Position Tracking

To map the patterns onto the body, one developer and I created a point cloud skeleton which generated anchor points using depth data captured by a Kinect sensor.

We then created a dynamic matte using a point cloud 3D map along with some post-processing tools, which continuously tracked the subjects body position and updated in real time.

Static to Animated Assets

I created a tiling module that took a input file, repeated it according to predefined parameters, and then funneled the tiled output into an animation module. Within the animation module, I developed a series of animations used as motion presets during the live performance.

Final Outcome

Music with Audience Contribution

The day of the performance, we projected a QR code on screen that prompted audience members to scan and color in patterns that would be featured in the performance. Audience members expressed how fun and engagingthe coloring experience was.

Our team quickly loaded audience patterns into TouchDesigner where all files cycled through a switch module. During the performance, all patterns passed through prebuilt animation presets, which were live projected onto the artist.

touchdesigner calibration interface

touchdesigner calibration tool

Conclusion

Different work pipelines

As the sole designer, I simultaneously created patterns, developed UI, learned TouchDesigner, and created animations, all while coordinating with a team and managing tight asset deadlines. Working closely and directly with each aspect of the project taught me a lot about development workflows, and live show planning.

Plan to be flexible

There were countless times our team had to pivot or work to find new solutions due to challenges or timeline constraints. This project required constant problem-solving, improvising, and communication.

A round of applause for EKUA and my amazing team!

Xin Yu Jiang
Jay Pruitt
Evelyn Zou

☆ More case studies!

Wegmans Smart Cart Project Thumbnail

Wegmans GoCart

UX Research

UI Design

Brand Identity

Path Pursuit thumbnail gif

Path Pursuit

UX Research

UI Design

Brand Identity