Natural and efficient interaction remains a critical challenge for virtual reality and augmented reality (VR/AR) systems. Vision-based gesture recognition suffers from high computational cost, sensitivity to lighting conditions, and privacy leakage concerns. Acoustic sensing provides an attractive alternative: by emitting inaudible high-frequency signals and capturing their reflections, channel impulse response (CIR) encodes how gestures perturb the acoustic field in a low-cost and user-transparent manner. However, existing CIR-based gesture recognition methods often rely on extensive training of models on large labeled datasets, making them unsuitable for few-shot VR scenarios. In this work, we propose the first framework that leverages large language models (LLMs) for CIR-based gesture recognition in VR/AR systems. Despite LLMs' strengths, it is non-trivial to achieve few-shot and zero-shot learning of CIR gestures due to their inconspicuous features. To tackle this challenge, we collect differential CIR rather than original CIR data. Moreover, we construct a real-world dataset collected from 10 participants performing 15 gestures across three categories (digits, letters, and shapes), with 10 repetitions each. We then conduct extensive experiments on this dataset using an LLM-adopted classifier. Results show that our LLM-based framework achieves accuracy comparable to classical machine learning baselines, while requiring no domain-specific retraining.

翻译：自然高效的交互仍是虚拟现实与增强现实（VR/AR）系统面临的关键挑战。基于视觉的手势识别存在计算成本高、对光照条件敏感及隐私泄露风险等问题。声学传感提供了一种极具吸引力的替代方案：通过发射不可闻的高频信号并捕获其反射信号，信道冲激响应（CIR）能以低成本且对用户透明的方式编码手势对声场的扰动特征。然而，现有基于CIR的手势识别方法通常依赖大规模标注数据集进行模型训练，难以适应小样本VR场景。本研究首次提出利用大语言模型（LLMs）实现VR/AR系统中基于CIR的手势识别框架。尽管LLMs具有强大能力，但由于CIR手势特征不显著，实现其小样本与零样本学习仍具挑战性。为此，我们采集差分CIR数据而非原始CIR数据。同时构建了包含10名参与者执行3类手势（数字、字母、图形）共15种动作的现实数据集，每类动作重复10次。基于该数据集，我们采用LLM适配分类器进行大量实验。结果表明：本框架在无需领域特定重训练的情况下，其识别准确率已达到经典机器学习基线方法的可比水平。