The potential for rapidly-evolving frontier artificial intelligence (AI) models, especially large language models (LLMs), to facilitate bioterrorism or access to biological weapons has generated significant policy, academic, and public concern. Both model developers and policymakers seek to quantify and mitigate any risk, with an important element of such efforts being the development of model benchmarks that can assess the biosecurity risk posed by a particular model. This paper discusses the pilot implementation of the Bacterial Biothreat Benchmark (B3) dataset. It is the third in a series of three papers describing an overall Biothreat Benchmark Generation (BBG) framework, with previous papers detailing the development of the B3 dataset. The pilot involved running the benchmarks through a sample frontier AI model, followed by human evaluation of model responses, and an applied risk analysis of the results along several dimensions. Overall, the pilot demonstrated that the B3 dataset offers a viable, nuanced method for rapidly assessing the biosecurity risk posed by a LLM, identifying the key sources of that risk and providing guidance for priority areas of mitigation priority.

翻译：快速演进的前沿人工智能（AI）模型，特别是大语言模型（LLMs），可能助长生物恐怖主义或促进生物武器获取，这已引发政策、学术界和公众的广泛关注。模型开发者和政策制定者均致力于量化并缓解相关风险，其中一项关键工作是开发能够评估特定模型所构成生物安全风险的模型基准。本文讨论了细菌生物威胁基准（B3）数据集的试点实施。这是描述整体生物威胁基准生成（BBG）框架的三篇系列论文中的第三篇，前两篇详细阐述了B3数据集的开发过程。试点工作包括通过一个前沿AI模型样本运行基准测试，随后对模型响应进行人工评估，并从多个维度对结果进行应用风险分析。总体而言，试点表明B3数据集提供了一种可行且细致的方法，能够快速评估LLM所构成的生物安全风险，识别风险的关键来源，并为优先缓解领域提供指导。