Evals for 3D CAD (www.emergentmind.com)

Researchers have introduced MUSE, a new benchmark designed to enhance Text-to-CAD generation, particularly for complex, editable boundary representation (B-Rep) assemblies. Traditional benchmarks have primarily focused on single-part CAD models and relied on geometric similarity metrics, which do not adequately assess critical aspects such as functionality, manufacturability, and assemblability. MUSE aims to fill this gap by offering a three-stage evaluation protocol that includes code checking, geometric validation, and alignment with design intent through a rubric-based visual LLM (VLM) judge, which has been validated against human annotations. This advancement is significant for the AI/ML community, particularly those working in industrial product design, because it moves away from mere shape matching to a more holistic evaluation of design quality. Initial experiments indicate that even the best-performing large language models struggle with the shift from executable code to valid geometry and ultimately to engineering-ready designs. By providing a structured evaluation framework, MUSE lays the groundwork for future developments in text-driven 3D generation, guiding AI systems toward producing not only geometrically accurate but also functionally viable and manufacturable designs. Access to the project’s datasets and leaderboard is available online, fostering further research and collaboration in this promising field.