StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation

Project Page | Paper

Overview

StyleSDF is a 3D-aware GAN, aimed at solving two main challenges: 1. High-resolution, view-consistent generation of the RGB images. 2. Generating detailed 3D shapes.

StyleSDF is trained only on single-view RGB data. The 3D geometry is learned implicitly with an SDF-based volume renderer.

Usage

You may choose to generate animal faces (trained on the AFHQ dataset) or human faces (trained on the FFHQ dataset), as well as select the number of identities (unique faces) and number of view angles.

If generate_model is set to True, the model outputs a grid of RGB images, where each row represents the angles generated from one identity. The model also outputs downloadable zipped files containing 3D meshes generated from a depth map, and meshes extracted with the Marching Cubes algorithm.

If generate_video is False, the model outputs a reconstructed 3D video and a depth video. Only one identity is processed.The process may take up to 10 minutes.

This code is the official PyTorch implementation of the paper:

StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation
Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, Ira Kemelmacher-Shlizerman
CVPR 2022
https://arxiv.org/pdf/2112.11427.pdf

Abstract

We introduce a high resolution, 3D-consistent image and shape generation technique which we call StyleSDF. Our method is trained on single-view RGB data only, and stands on the shoulders of StyleGAN2 for image generation, while solving two main challenges in 3D-aware GANs: 1) high-resolution, view-consistent generation of the RGB images, and 2) detailed 3D shape. We achieve this by merging a SDF-based 3D representation with a style-based 2D generator. Our 3D implicit network renders low-resolution feature maps, from which the style-based network generates view-consistent, 1024×1024 images. Notably, our SDFbased 3D modeling defines detailed 3D surfaces, leading to consistent volume rendering. Our method shows higher quality results compared to state of the art in terms of visual and geometric quality.

Citation

If you use this code for your research, please cite our paper.

@article{orel2021stylesdf,
  title={Style{SDF}: {H}igh-{R}esolution {3D}-{C}onsistent {I}mage and {G}eometry {G}eneration},
  author={Or-El, Roy and
          Luo, Xuan and
          Shan, Mengyi and
          Shechtman, Eli and
          Park, Jeong Joon and
          Kemelmacher-Shlizerman, Ira},
  journal={arXiv preprint arXiv:2112.11427},
  year={2021}
}

Acknowledgments

This code is inspired by rosinality’s StyleGAN2-PyTorch and Yen-Chen Lin’s NeRF-pytorch.