Physically Controllable Relighting of Photographs (yaksoy.github.io)

A groundbreaking photograph relighting technique has been introduced, enabling users to exert explicit control over illumination in images, similar to processes found in computer-generated (CG) environments. This method integrates mid-level computer vision, physically-based rendering, and neural rendering, allowing for the insertion and manipulation of diverse light sources such as spotlights and environmental lights within a scene. The approach utilizes a self-supervised training process leveraging differentiable rendering, effectively training a neural renderer on large collections of real-world photographs to enhance generalization capabilities. This development is significant for the AI/ML community as it bridges the gap between traditional photo editing and the advanced capabilities of 3D rendering tools. By inferring a colored mesh representation of the scene and employing a path-tracing engine to simulate new lighting conditions, this methodology not only provides precise control over lighting but also ensures photorealistic results. The use of an extended intrinsic model paves the way for more sophisticated editing tasks, such as specularity removal and per-pixel white balancing, further pushing the boundaries of computational photography and image editing capabilities in real-world scenarios.