Sean Fanello, Julien Valentin, Jonathan Taylor, Christoph Rhemann, Adarsh Kowdle, Jürgen Sturm,

Christine Kaeser-Chen, Pavel Pidlypenskyi, Rohit Pandey, Sameh Khamis,

David Kim, Mingsong Dou, Kaiwen Guo, Danhang Tang, Shahram Izadi

Augmented Perception - Google


The emergence of virtual and augmented reality has increased the demand of robust systems for 3D capture, reconstruction and understanding. Designing such systems involves developing high quality sensors and efficient algorithms that can leverage new and existing technologies. To this end, we design depth sensors with two properties that drastically simplify the problem of fusing incomplete sensor data. First, an UltraFast (~1000 fps) depth stream that significantly reduces the frame to frame motion. Second, by allowing multiple sensors to be easily combined (without interference) we can eliminate occlusions. As a result, we have developed a wide range of efficient algorithms for reconstruction, tracking and understanding that are designed to work with this technology. In this tutorial we will walk the audience through the ins and outs of building such a stack from the ground up. From building such a sensor to applications for mixed reality with particular emphasis on high speed 3D capture systems.

When and Where

Friday June 22nd 2018
Ball A - Calvin L. Rampton Salt Palace Convention Center, Salt Lake City


9:00 - 9:10
Shahram Izadi
AP - Google
Session 1:
Depth Sensors & 3D Capture
9:10 - 9:40
Depth Sensors and Algorithms: What, When, Where
Adarsh Kowdle
AP - Google
9:40 - 10:10
Triangulation Methods: From Theory to Mobile
Christoph Rhemann
AP - Google
10:10 - 10:30
UltraFast Machine Learning for Triangulation Methods
Sean Fanello
AP - Google
10:30 - 11:00
Coffee Break
11:00 - 11:15
Deep Learning to Break 1/30th Subpixel Precision
Sameh Khamis
AP - Google
11:15 - 11:30
Active Stereo Net: Self-Supervised End-to-End Training for Active Stereo Systems
Yinda Zhang
Princetone University / AP - Google
Session 2:
Tracking & Relocalization
11:30 - 12:00
Non-Linear Optimization Methods
Jonathan Taylor
AP - Google
12:00 - 13:00
Lunch Break
13:00 - 13:45
Localization and Mapping - ARCore
Konstantine Tsotsos
Zack Moratto
ARCore - Google
13:45 - 14:00
Machine Learning for Camera Relocalization
Julien Valentin
AP - Google
Session 3:
World Understanding
14:00 - 14:15
3D Scene Understanding
Juergen Sturm
AP - Google
14:15 - 14:30
Semantic Labeling
Thomas Funkhouser
Princetone University / AP - Google
Session 4:
Non Rigid Tracking & Reconstruction
14:30 - 14:50
Parametric Tracking
Anastasia Tkach
AP - Google
14:50 - 15:30
Non Rigid Reconstruction of Humans
Mingsong Dou
Kaiwen Guo
AP - Google
15:30 - 16:00
Coffee Break
Session 5:
Machine Learning and Interactive Applications
16:00 - 16:15
LookinGood - Free Viewpoint Volumetric Videos
Rohit Pandey
AP - Google
16:15 - 16:30
Fast Machine Learning on Embedded Devices
Christine Kaeser-Chen
Mobile Vision - Google
16:30 - 16:15
Digital AR
David Kim
AP - Google

Please contact Shahram Izadi or Sean Fanello if you have any questions.