GaussianGPT: Generating 3D scenes autoregressively with Gaussian Splatting (github.com)

Researchers have introduced GaussianGPT, a groundbreaking transformer-based model that enables autoregressive generation of 3D scenes using Gaussian splatting techniques. Unlike traditional diffusion-based models that refine scenes holistically, GaussianGPT generates scenes sequentially by directly predicting 3D Gaussian tokens, which allows for greater flexibility in completion, outpainting, and controllable sampling. The model employs a sparse 3D convolutional autoencoder to convert Gaussian primitives into a discrete latent grid, serialized for processing by a causal transformer enhanced with 3D rotary positional embeddings. This innovation is significant for the AI/ML community as it offers a new paradigm for 3D generative modeling that emphasizes the compositional strengths and scalability of autoregressive techniques. The capability of GaussianGPT to create context-aware and controllable 3D environments positions it as a potent alternative to existing methods, potentially influencing applications in virtual reality, simulation, and game design. The project includes comprehensive code, pretrained models, and detailed training instructions, making it accessible for further research and development in the field of 3D scene generation.