VoLux-GAN: A Generative Model for 3D Face Synthesis with HDRI Relighting


Feitong Tan1,2      Sean Fanello1      Abhimitra Meka1      Sergio Orts-Escolano1      Danhang Tang1      Rohit Pandey1     
Jonathan Taylor1      Ping Tan2      Yinda Zhang1

1   Google                                 2    Simon Fraser University
                                       



Albedo
Relit Image
Normals
Diffuse Shading
specular Shading



Abstract


We propose VoLux-GAN, a generative framework to synthesize 3D-aware faces with convincing relighting. Our main contribution is a volumetric HDRI relighting method that can efficiently accumulate albedo, diffuse and specular lighting contributions along each 3D ray for any desired HDR environmental map. Additionally, we show the importance of supervising the image decomposition process using multiple discriminators. In particular, we propose a data augmentation technique that leverages recent advances in single image portrait relighting to enforce consistent geometry, albedo, diffuse and specular components. Multiple experiments and comparisons with other generative frameworks show how our model is a step forward towards photorealistic relightable generative models.




Paper



[arXiv]     [GitHub] (Coming Soon)    

Download the paper (2.9M) here.

Download the supplementary (3.1M) here.


Citation

BibTeX, 1 KB

@misc{tan2022voluxgan,
            title={VoLux-GAN: A Generative Model for 3D Face Synthesis with HDRI Relighting}, 
            author={Feitong Tan and Sean Fanello and Abhimitra Meka and Sergio Orts-Escolano and Danhang Tang and Rohit Pandey and Jonathan Taylor and Ping Tan and Yinda Zhang},
            year={2022},
            eprint={2201.04873},
            archivePrefix={arXiv},
            primaryClass={cs.CV}
      }
        


More Results






Albedo
Relit Image
Normals
Diffuse Shading
specular Shading



Relighting Compared with a Light Stage




Environmental Relighting of a real person using a Light Stage
Environmental Relighting of a generated identity using VoLux-GAN
Directional Relighting of a generated identity using ShadeGAN

We show a comparison of our relighting method with environmental relighting of a real person using a Light Stage, and state-of-the-art directional relighting method of ShadeGAN.


Relighting a Single Generated Identity Under Different Environments



Environmental 1
Environmental 2
Environmental 3

We show relighting results of our method for a single generated identity under different rotating environment maps.


Relighting Multiple Generated Identities Under Same Environment



Identity 1
Identity 2
Identity 3

We show relighting results of our method for randomly generated identities under the same rotating environment.


Relighting Multiple Generated Identities Under Different Environments



Identity 1
Identity 2
Identity 3

We show relighting results of our method for randomly generated identities under different rotating environment maps.


View Synthesis of Different Generated Identities



Identity 1
Identity 2
Identity 3

We show view synthesis results of our method for randomly generated identities under a fixed environment map.