A live interactive projection mapping performance

ekua motion graphic

EKUA: To Get To The Other Side

EKUA: To Get To The Other Side is an interactive music experience that bridges live performance, projection art, and customized audience contribution.

𖤐 Role

UI Design
Visual Identity
Motion Design

✐ Tools

Figma
Illustrator
TouchDesigner

⏱︎ Duration

12 weeks

𖠋 Team

1 Designer
3 Developers

ekua singing on live performance date

Background

An interdisciplinary collaboration between design, technology, and performance art

Our team was paired with EKUA, a local Rochester, NY musician, to co-create an interactive experience in support of her latest project, To Get To The Other Side.

After an initial introduction to EKUA’s artistic practice, we developed three potential concepts inspired by her values, our technical skill sets, and the project timeline. Our task was to design an experience that would harmoniously marry EKUA’s music with audience interaction.

Following several collaborative meetings, we decided to pursue a mixed-reality live performance that utilized projections to map and animate audience customized, dynamic patterns onto the artist’s body and the surrounding environment in sync with the music.

Client Background

Who is EKUA?

Natalia 'EKUA' France is a Ghanaian-American artist whose work exists at the intersection of fashion, visual art, beauty, dance, and music. Her practice is influenced by her cultural heritage, spirituality, and contemporary expression.

EKUA’s music blends Afrobeat, R&B, soul, and pop; she is inspired by Bob Marley, Lauryn Hill, Tems, Rihanna, and many others. Through her music, she encourages listeners to feel deeply and heal courageously.

Research

Developing the Experience

Before defining core components of our experience, our team agreed that we wanted to create an experience that immersed users in a sensorily rich environment—whether through visuals, sound, or both.

EKUA expressed interest in integrating fashion and dance, which guided our research and ideation. Below are two experiences that were major inspirations for our project.

Franck Sorbier F/W 2013 Dress Projections

Franck Sorbier’s Fall/Winter 2013 collection uses projection mapping to animate a white gown creating a dynamic merging of fashion and technology.

Features

Dynamic Dress Animation

Variable Environments

Quick Transitions

Customizable

Takeaways

Projection mapping amplifies both the model and the environment.

Projection creates opportunity for experimental motion.

teamLab's Sketch Ocean

Sketch Ocean was an interactive installation that had participants color their own fish illustrations, which were then digitally brought to life, and released into a virtual aquarium environment.

Features

Personalization

Real-life Tactile Input

Real-time Animation

Community Interaction

Takeaways

Users are able to contribute personal and meaningful inputs.

Users can interact with other users' inputs

teamLabs interactive exhibit

Goals

Center the music

Although creating an interactive experience was our priority, we wanted to ensure that it never overshadowed the music or the performance itself.

Connect audience and artist

The performance experience should spark curiosity and excitement for EKUA’s music and new project.

Create meaningful interaction

Audience interaction shouldn’t feel gimmicky; it should create meaningful points of engagement. Every aspect of the interaction should be enjoyable, engaging, and fun.

Project Management

Team Oriented Work Pipeline

As the team lead for this project, I collaborated with my team to define the key tasks required to bring our interactive performance concept to life. I designed the user interface and led the visual direction, planned event user experience, created TouchDesigner animations, and acted as the primary liaison between our team and the artist.

Two developers focused on web development and TouchDesigner integration, while another developer built body-tracking motion capture modules in TouchDesigner. The following outlines the main tasks our team focused on throughout the project.

1. Interactive coloring app

Develop a web-based application that users can easily access on the day of the performance.

2. Static and moving visual assets

Create animated visuals for the live performance and design branded promotional material.

3. TouchDesigner integration

Create a streamlined pipeline that animates customized user input and projects it live onto the performer.

4. Event logistics

Plan all event logistics including equipment needs, production schedule, and user experience.

User Flow

Live Event Planning

I created a simple event flow to better estimate how much time we’ll need audience members to participate. The time it takes them to open the app and complete a round of coloring will directly affect how long it takes to transfer the files and load all necessary assets into TouchDesigner on the day of the performance.

ekua event user flow

Design

User Interface Design

When designing the web-based coloring app, the primary goal was to create a simple flow that encouraged users to be able to create as many patterns as possible with minimal effort on the performance day.

Moodboard

When developing imagery for the projections, EKUA expressed an interest in drawing inspiration from traditional Ghanaian textiles and West African spirituality. I created a mood board with EKUA to establish color and form themes. We decided on visuals centered around vibrant patterning.

ekua interactive coloring application

Simple Patterns

When creating patterns, it became clear that the simplest patterns, were the most effective for our purposes.

After some user testing, we found that users applying color to more complex patterns took significantly more time, as they spent longer contemplating their choices, which slowed overall output.

pattern 1

pattern 2

pattern 3

pattern 4

pattern 5

pattern 6

Live Body Position Tracking

I collaborated with one of our developers to create a point cloud skeleton in TouchDesigner, which generated anchor points from depth data captured by a Kinect sensor.

To map the patterns onto the body, we used a point cloud 3D map along with some post-processing tools to create a dynamic matte. This matte outline updated in real time, continuously tracking the subject’s body as their position fluctuated.

Static to Animated Assets

I created a tiling module in TouchDesigner that could take an input file and repeat it according to predefined parameters. Using the resulting tiled sequences, I then developed a series of simple animations, which were used as presets during the live performance.

Final Outcome

Music with Audience Contribution

The day of the performance, we displayed a QR code on screen that prompted audience members to color in patterns that would be featured in the performance. Audience members expressed how fun coloring was.

Our team quickly loaded audience patterns into TouchDesigner where all files cycled through a switch module. During the performance, all patterns passed through prebuilt animation presets, which were live projected onto the artist.

touchdesigner calibration interface

touchdesigner calibration tool

Conclusion

Different work pipelines

As the sole designer, I simultaneously created patterns, developed UI, learned TouchDesigner, and created animations, all while coordinating with a team and managing tight asset deadlines. Working closely and directly with each aspect of the project taught me a lot about development workflows, and live show planning.

Plan to be flexible

There were countless times our team had to pivot or work to find new solutions due to challenges or timeline constraints. This project required constant problem-solving, improvising, and communication.

A round of applause for EKUA and my amazing team!

Xin Yu Jiang
Jay Pruitt
Evelyn Zou

☆ More case studies!

Wegmans Smart Cart Project Thumbnail

Wegmans GoCart—UX/UI Design

UX Research

UI Design

Brand Identity

Path Pursuit thumbnail gif

Path Pursuit—Interactive Exhibit Design

UX Research

UI Design

Brand Identity