OctLLM – Octrees as an Explicit 3D Language (plurato.github.io)

OctLLM has been introduced as a groundbreaking approach in 3D language models, effectively addressing key limitations of existing systems which often lose spatial structure or require resource-intensive fine-tuning for 3D capabilities. It employs a novel method of incorporating geometry through Sparse Octrees using occupancy tokens tied to 3D coordinates and depth, which allows for efficient position-aware mask modeling and improved understanding of 3D data without bloating the model size. By randomly omitting certain nodes within the octree structure, OctLLM maintains essential shape details while significantly reducing complexity. Moreover, OctLLM distinguishes itself by integrating new 3D capacity without altering the pretrained language model’s parameters. It features independent trainable branches for 3D mesh tokens while preserving the frozen text-image pathway, allowing for seamless interaction via shared self-attention. This innovative architecture leads to superior performance, establishing new benchmarks by reducing the image-to-3D Fréchet Inception Distance (FID) by 17.4% and enhancing render-grounded captioning by 28.7 points compared to previous models. As a result, OctLLM not only advances the field of multimodal learning but also sets a precedent for efficient cross-modal interactions while maintaining general language capability.