Visual artist Sally Coulden is transformed into a beautifully rendered hologram, presented in AR for anyone to watch her painting process from home.

Abstracted is a new work commissioned with support from the UK Arts Council which sees artist Sally Coulden’s creative process brought to life through an augmented reality app. Zubr created a point cloud renderer and built the AR interface and user experience to power this exciting iOS app, released during Lockdown to invite audiences to experience Sally’s creative works from wherever they are.

Capturing the process

Collaborators Scott Fletcher and Benjamin Walbrook worked alongside Sally during her creative process, capturing a 10-hour timelapse of an original canvas painting. Furthermore, a motion capture session provided the true-to-life physical movements of the artist as she painted, deliberated and admired – including everything from delicate and hesitant mark-making to confident and powerful strokes.

Motion Capture in Augmented Reality

The mo-cap data was applied to a 3D avatar, allowing the realtime movements to be played back in the Unity engine. Motion capture data is extremely versatile and surprisingly lightweight, especially compared to alternatives such as volumetric video – making it the perfect basis for a well-optimised public app. The Zubr team prepared the material to be presented as a miniature AR scene which can be placed by users on any surface.

Point Cloud rendering

We took inspiration from our previous projects featuring point cloud holographic visual effects to create a mesmerising style for Sally’s virtual persona. The underlying 3D avatar model is still present, but invisible – acting as the emitter of the stardust-like particles which create beautiful trails and patterns as they follow Sally’s movements around the canvas.

Available on the App Store

The finished work is available for anyone with an ARKit-compatible Apple device to download from the App Store. Upon launching the AR Experience, you are free to choose where to place the 3D scene before Sally’s creative process begins. You can move around and manipulate the scene as much as you want during the 5-minute experience.