Several studies showed that misuses of cryptographic APIs are common in real-world code (e.g., Apache projects and Android apps). There exist several open-sourced and commercial security tools that automatically screen Java programs to detect misuses. To compare their accuracy and security guarantees, we develop two comprehensive benchmarks named CryptoAPI-Bench and ApacheCryptoAPI-Bench. CryptoAPI-Bench consists of 181 unit test cases that cover basic cases, as well as complex cases, including interprocedural, field sensitive, multiple class test cases, and path sensitive data flow of misuse cases. The benchmark also includes correct cases for testing false-positive rates. The ApacheCryptoAPI-Bench consists of 121 cryptographic cases from 10 Apache projects. We evaluate four tools, namely, SpotBugs, CryptoGuard, CrySL, and Coverity using both benchmarks. We present their performance and comparative analysis. The ApacheCryptoAPI-Bench also examines the scalability of the tools. Our benchmarks are useful for advancing state-of-the-art solutions in the space of misuse detection.

翻译：几项研究表明,滥用加密API在现实世界代码中很常见(例如,阿帕奇项目和Android Apps)。有一些开放源码和商业安全工具自动筛选爪哇方案以检测滥用情况。为了比较其准确性和安全保障,我们制定了两个综合基准,称为CryptoAPI-Bench和ApacheCryptoAPI-Bench。CryptoAPI-Bench由181个单位测试案例组成,涵盖基本案例和复杂案例,包括程序间、实地敏感、多级测试案例和滥用案件路径敏感数据流。基准还包括测试假阳性率的正确案例。ApacheCryptoAPI-Bench由10个阿帕奇项目的121个加密案例组成。我们用两个基准评估了四个工具,即SpotBugs、CryptoGuard、CrySL和Coverty。我们介绍了它们的业绩和比较分析。ApacheCrypto API-Bench还审查了工具的可测量性。我们的基准对于推进空间探测滥用的状态解决方案很有帮助。