Encoder-free architectures have been preliminarily explored in the 2D Large Multimodal Models (LMMs), yet it remains an open question whether they can be effectively applied to 3D understanding scenarios. In this paper, we present the first comprehensive investigation into the potential of encoder-free architectures to alleviate the challenges of encoder-based 3D LMMs. These long-standing challenges include the failure to adapt to varying point cloud resolutions during inference and the point features from the encoder not meeting the semantic needs of Large Language Models (LLMs). We identify key aspects for 3D LMMs to remove the pre-trained encoder and enable the LLM to assume the role of the 3D encoder: 1) We propose the LLM-embedded Semantic Encoding strategy in the pre-training stage, exploring the effects of various point cloud self-supervised losses. And we present the Hybrid Semantic Loss to extract high-level semantics. 2) We introduce the Hierarchical Geometry Aggregation strategy in the instruction tuning stage. This incorporates inductive bias into the LLM layers to focus on the local details of the point clouds. To the end, we present the first Encoder-free 3D LMM, ENEL. Our 7B model rivals the state-of-the-art model, PointLLM-PiSA-13B, achieving 57.91%, 61.0%, and 55.20% on the classification, captioning, and VQA tasks, respectively. Our results show that the encoder-free architecture is highly promising for replacing encoder-based architectures in the field of 3D understanding. The code is released at https://github.com/Ivan-Tang-3D/ENEL

翻译：无编码器架构已在二维大语言模型中进行了初步探索，但其能否有效应用于三维理解场景仍是一个开放性问题。本文首次全面研究了无编码器架构在缓解基于编码器的三维大语言模型所面临挑战方面的潜力。这些长期存在的挑战包括：在推理过程中无法适应变化的点云分辨率，以及编码器提取的点特征无法满足大语言模型的语义需求。我们确定了三维大语言模型移除预训练编码器并让大语言模型承担三维编码器角色的关键方面：1）在预训练阶段，我们提出了大语言模型嵌入的语义编码策略，探索了多种点云自监督损失函数的效果，并提出了混合语义损失以提取高层语义。2）在指令微调阶段，我们引入了分层几何聚合策略，将归纳偏置融入大语言模型层，使其关注点云的局部细节。最终，我们提出了首个无编码器三维大语言模型ENEL。我们的70亿参数模型在分类、描述生成和视觉问答任务上分别达到了57.91%、61.0%和55.20%的性能，与当前最先进的模型PointLLM-PiSA-130亿参数模型相媲美。我们的结果表明，无编码器架构在三维理解领域替代基于编码器的架构具有巨大潜力。代码发布于https://github.com/Ivan-Tang-3D/ENEL