Renderformer V2 (Full neural rendering) (renderformer.github.io)

RenderFormer-V2 has been introduced as a groundbreaking transformer-based neural rendering model that enhances the capabilities of modern physics-based renderers. This model can simultaneously manage various light transport effects—including refraction, volumetric scattering, and diverse materials—without the need for scene-specific training. The significant advancement lies in its two-stage architecture: a view-independent stage that utilizes a novel combination of windowed attention mechanisms to resolve complex scene details, followed by a view-dependent stage that converts these details into high-quality image pixels. RenderFormer-V2 boasts a capability to handle over 128K primitives at higher resolutions (up to 2048x2048) while maintaining rendering efficiency. The innovations in RenderFormer-V2 include a new representation for materials based on a 9-D latent appearance space and a more effective attention mechanism that scales the rendering process. This architecture allows for a more flexible and inclusive representation of scene complexities, accommodating a broader range of lighting and material types. Notably, the model was trained on a substantial dataset of approximately 70 TB over a lengthy period, allowing it to generalize well to unseen materials and configurations. Its performance indicates notable improvements over its predecessor, RenderFormer-V1, particularly in rendering fidelity and efficiency, making it a substantial development for the AI/ML community focused on computer graphics and rendering technologies.