Sean Fanello, Christoph Rhemann, Graham Fyffe, Jonathan Taylor, Sofien Bouaziz, Rohit Pandey, Paul Debevec, Shahram Izadi



Volumetric (4D) performance capture is fundamental for AR/VR content generation. Designing a volumetric capture pipeline involves developing a full computer vision stack: from novel sensors to efficient volumetric reconstruction algorithms, compression and rendering. To this end, we leverage a combination of active sensors with traditional photometric stereo methods. As a result, we have developed an unique pipeline for reconstruction, tracking and texturing of humans in 4D.

In this tutorial we will walk the attendee through the ins and outs of building such a system from the ground up.

First, we will consider the hardware design choices for cameras, sensors, lighting, and depth estimation algorithms.

In the second part we will cover reconstruction and tracking techniques for people. We will review state of the art algorithms to generate high quality meshes and volumetric representations.

In the third part we will focus on photometric stereo, texturing, relightability and rendering: we will detail the state of the art algorithms together with our choices.

Finally we will discuss some applications and capabilities that 3D capture technologies enable. We will put emphasis on virtual and augmented reality scenarios and highlight recent trends in machine learning that aim at replacing traditional graphics pipelines.

When and Where

June 16th - Long Beach Convention Center in Long Beach, Room 201A


9:00 - 9:30
[Keynote] Digitizing Humans
Shahram Izadi
Morning Session:
Volumetric Capture of Humans
9:30 - 10:00
High Quality Depth Sensors for Volumetric Capture
Christoph Rhemann, Adarsh Kowdle
10:00 - 10:30
Depth Estimation via Triangulation: Basics, Challenges and Algorithms
Sergio Orts Escolano
10:30 - 10:45
Coffee Break
10:45 - 11:15
Depth Estimation in the Age of Deep Learning
Yinda Zhang, Christian Haene, Sameh Khamis
11:15 - 11:40
Multiview Human Pose Estimation via Deep Learning
Rohit Pandey, Anastasia Tkach
11:40 - 12:30
Detect, Reconstruct, Track and Parameterize Humans
Sean Fanello, Kaiwen Guo
12:30 - 13:30
Lunch Break
Afternoon Session:
Light Stage for High Quality Renderings
13:30 - 14:00
[Keynote] A Light Stage for (almost) Every Application
Paul Debevec
14:00 - 14:25
The Lightstage Hardware
Xueming Yu
14:25 - 15:00
Relighting Still Images and Videos
Graham Fyffe
15:00 - 15:20
Relightable Volumetric Capture of Humans
Graham Fyffe
15:20 - 15:40
Single Image Portrait Relighting
Tiancheng Sun
15:40 - 16:00
Coffee Break
16:00 - 16:20
Neural rendering for Performance Capture
Rohit Pandey
16:20 - 16:40
Neural Rerendering in the Wild
Moustafa Meshry
University of Maryland
16:40 - 17:00
Deep Reflectance Fields
Abhimitra Meka

Please contact Shahram Izadi or Sean Fanello if you have any questions.