We will explore generative AI with a focus on 3D content creation and editing. You will build up your background in 3D computer graphics and 3D computer vision and conduct research in the following themes including neural radiance fields, diffusion models, or applied projects.
1. Neural radiance fields
Project 1.1: Real-time neural radiance fields. We will explore lightweight scene representation, rendering, and training procedures to optimize neural radiance fields for applications with real-time performance.
Project 1.2: Improved sampling for neural radiance field rendering. We will revisit ray sampling and volume rendering techniques in existing neural radiance fields and improve the rendering procedure with state-of-the-art physically based rendering techniques.
Project 1.3: Immersive neural radiance fields. We will explore and develop robust representation and rendering techniques for panoramic 360-degree neural radiance fields that provide immersive experience.
2. Diffusion models
We will explore content creation using GANs and diffusion models with multi-modality input and output including text-to-image and text-to-3D content generation with a focus on improving the quality of scene geometry and appearance.
Project 2.1: 3D diffusion models. We will explore the construction of diffusion models with 3D datasets such as ShapeNet and Objaverse and build a large-scale diffusion model for 3D content creation.
Project 2.2: Neural radiance field editing. We will use generative models in both the 2D and 3D domains to generate content to support user interaction to manipulate content captured by neural radiance fields. We can also aim at different editing tasks including but not limited to object insertion, removal, transformation, texturing, relighting, and animation.
Project 2.3: Language-driven 3D content creation. We will research text-to-3D generation methods such as the recent DreamFusion and ProlificDreamer with a focus on improving the realism of the generated content.
3. Applications
Project 3.1: Game development with generative AI. We will build a real-time 3D game that only uses generative AI techniques such as neural rendering, generative language/image/3D models for its asset creation. The goal is to revisit the traditional workflow and adopt generative AI techniques to improve and enrich the in-game assets. Using AR/VR is a plus for this project.
Project 3.2: Generative 3D content creation tools. We will build a next-generation 3D authoring tool that adopts state-of-the-art generative 2D and 3D models for interactive content creation.
For applied projects, it is required that you have a strong background in software engineering and relevant programming skills in C++ and Python.
Further notes
I am super-interested in working on projects that can potentially submitted to top-tier computer vision and computer graphics venues such as CVPR, ECCV, ICCV, SIGGRAPH, and Eurographics. You are a perfect match if you are highly interested in pursuing research, either in academia or industry, as your long-term career.
It is recommended that you have either taken computer graphics, computer vision, or machine learning modules with a strong coding background in Python or C++. Some experience with Pytorch, Unity, Unreal Engine, or Blender is a good plus.
If you have your own idea in 3D computer graphics and 3D computer vision in mind, you are welcome to discuss with me to develop this idea into a research project.