Measuring biodiversity is crucial for understanding ecosystem health. While prior works have developed machine learning models for taxonomic classification of photographic images and DNA separately, in this work, we introduce a multimodal approach combining both, using CLIP-style contrastive learning to align images, barcode DNA, and text-based representations of taxonomic labels in a unified embedding space. This allows for accurate classification of both known and unknown insect species without task-specific fine-tuning, leveraging contrastive learning for the first time to fuse barcode DNA and image data. Our method surpasses previous single-modality approaches in accuracy by over 8% on zero-shot learning tasks, showcasing its effectiveness in biodiversity studies.

翻译：测量生物多样性对于理解生态系统健康至关重要。尽管先前的研究已分别开发了用于照片图像和DNA分类的机器学习模型，但本研究提出了一种多模态方法，将两者结合，利用CLIP风格的对比学习在统一的嵌入空间中对齐图像、条形码DNA以及基于文本的分类标签表示。该方法首次利用对比学习融合条形码DNA与图像数据，无需任务特定微调即可实现对已知和未知昆虫物种的准确分类。在零样本学习任务中，我们的方法在准确率上超越了以往的单模态方法超过8%，证明了其在生物多样性研究中的有效性。