Large language models (LLMs) have been widely evaluated on macro-scale geographic tasks, such as global factual recall, event summarization, and regional reasoning. Yet, their ability to handle hyper-local knowledge remains poorly understood. This gap is increasingly consequential as real-world applications, from civic platforms to community journalism, demand AI systems that can reason about neighborhood-specific dynamics, cultural narratives, and local governance. Existing benchmarks fall short in capturing this complexity, often relying on coarse-grained data or isolated references. We present LocalBench, the first benchmark designed to systematically evaluate LLMs on county-level local knowledge across the United States. Grounded in the Localness Conceptual Framework, LocalBench includes 14,782 validated question-answer pairs across 526 U.S. counties in 49 states, integrating diverse sources such as Census statistics, local subreddit discourse, and regional news. It spans physical, cognitive, and relational dimensions of locality. Using LocalBench, we evaluate 13 state-of-the-art LLMs under both closed-book and web-augmented settings. Our findings reveal critical limitations: even the best-performing models reach only 56.8% accuracy on narrative-style questions and perform below 15.5% on numerical reasoning. Moreover, larger model size and web augmentation do not guarantee better performance, for example, search improves Gemini's accuracy by +13.6%, but reduces GPT-series performance by -11.4%. These results underscore the urgent need for language models that can support equitable, place-aware AI systems: capable of engaging with the diverse, fine-grained realities of local communities across geographic and cultural contexts.

翻译：大型语言模型（LLMs）已在宏观地理任务（如全球事实回忆、事件总结和区域推理）中得到广泛评估。然而，其处理超本地知识的能力仍鲜为人知。随着现实应用（从公民平台到社区新闻）对能够推理社区特定动态、文化叙事和地方治理的人工智能系统的需求日益增长，这一空白愈发重要。现有基准测试在捕捉此类复杂性方面存在不足，往往依赖粗粒度数据或孤立参考。本文提出LocalBench，这是首个旨在系统评估LLMs在美国县级本地知识能力的基准测试。基于本地性概念框架，LocalBench包含来自美国49个州526个县的14,782个经过验证的问答对，整合了人口普查统计数据、本地Reddit论坛讨论和区域新闻等多种来源，涵盖地方的物理、认知和关系维度。利用LocalBench，我们在闭卷和网络增强两种设置下评估了13个最先进的大型语言模型。研究结果揭示了关键局限性：即使表现最佳的模型在叙事式问题上的准确率仅为56.8%，在数值推理任务中表现低于15.5%。此外，更大的模型规模和网络增强并不能保证更好的性能，例如，搜索使Gemini的准确率提升+13.6%，但使GPT系列模型性能下降-11.4%。这些结果突显了迫切需要能够支持公平、具备地域意识的人工智能系统的语言模型：能够应对不同地理和文化背景下本地社区多样且细粒度的现实情况。