NaTex: Seamless Texture Generation as Latent Color Diffusion (natex-ldm.github.io)

NaTex introduces a new paradigm for 3D texture synthesis: latent color diffusion that directly predicts RGB for 3D coordinates. The method pairs a color point-cloud VAE (inspired by 3DShape2VecSet but operating on color data) with a diffusion transformer (DiT). A dual-branch VAE embeds geometry tokens alongside color tokens to give strong geometric guidance during latent compression, achieving over 80× compression so diffusion can run efficiently in latent space. The DiT is multi-control — it accepts image, geometry, or initial-color controls via pairwise conditional geometry tokens implemented through positional embeddings and channel-wise concatenation — enabling precise surface hints and global context modeling for consistent, 3D-aware texture generation. NaTex addresses core MVD challenges — occlusion in multi-view texturing, fine-grained alignment to geometry, and multi-view consistency — and outperforms state-of-the-art methods on alignment, coherence, and occlusion handling using the same input image and geometry. Practical benefits include rapid, robust texture refinement and inpainting (completed in about five diffusion steps), material-attribute extension (roughness/metallic), and even training-free image-conditioned part segmentation by feeding 2D masks into the model. Overall, NaTex offers a scalable, flexible pipeline for high-fidelity, geometry-consistent textures that integrates cleanly into asset workflows and broadens latent-diffusion applications for 3D content creation.