Advances in 3D generative AI have enabled the creation of physical objects from text prompts, but challenges remain in creating objects involving multiple component types. We present a pipeline that integrates 3D generative AI with vision-language models (VLMs) to enable the robotic assembly of multi-component objects from natural language. Our method leverages VLMs for zero-shot, multi-modal reasoning about geometry and functionality to decompose AI-generated meshes into multi-component 3D models using predefined structural and panel components. We demonstrate that a VLM is capable of determining which mesh regions need panel components in addition to structural components, based on object functionality. Evaluation across test objects shows that users preferred the VLM-generated assignments 90.6% of the time, compared to 59.4% for rule-based and 2.5% for random assignment. Lastly, the system allows users to refine component assignments through conversational feedback, enabling greater human control and agency in making physical objects with generative AI and robotics.

翻译：3D生成式人工智能的进展已实现从文本提示生成物理对象，但在涉及多种组件类型的物体创建方面仍存在挑战。本文提出一种将3D生成式人工智能与视觉语言模型（VLMs）相结合的流程，实现从自然语言描述到多组件物体的机器人装配。该方法利用VLMs进行零样本、多模态的几何与功能推理，将AI生成的网格模型分解为使用预定义结构组件与面板组件的多组件3D模型。我们证明VLM能够根据物体功能，在结构组件之外判断哪些网格区域需要面板组件。在测试对象上的评估表明，用户对VLM生成的组件分配方案的偏好率达90.6%，而基于规则的方案为59.4%，随机分配方案仅为2.5%。最后，该系统允许用户通过对话反馈优化组件分配，从而在利用生成式人工智能与机器人技术制造物理对象时赋予人类更强的控制权与自主性。