Context: Mining software repositories is a popular means to gain insights into a software project's evolution, monitor project health, support decisions and derive best practices. Tools supporting the mining process are commonly applied by researchers and practitioners, but their limitations and agreement are often not well understood. Objective: This study investigates some threats to validity in complex tool pipelines for evolutionary software analyses and evaluates the tools' agreement in terms of data, study outcomes and conclusions for the same research questions. Method: We conduct a lightweight literature review to select three studies on collaboration and coordination, software maintenance and software quality from high-ranked venues, which we formally replicate with four independent, systematically selected mining tools to quantitatively and qualitatively compare the extracted data, analysis results and conclusions. Results: We find that numerous technical details in tool design and implementation accumulate along the complex mining pipelines and can cause substantial differences in the extracted baseline data, its derivatives, subsequent results of statistical analyses and, under specific circumstances, conclusions. Conclusions: Users must carefully choose tools and evaluate their limitations to assess the scope of validity in an adequate way. Reusing tools is recommended. Researchers and tool authors can promote reusability and help reducing uncertainties by reproduction packages and comparative studies following our approach.

翻译：背景：挖掘软件仓库是获取软件项目演化洞见、监控项目健康状况、支持决策制定以及推导最佳实践的常用方法。支持挖掘过程的工具常被研究人员和实践者使用，但其局限性与一致性往往未被充分理解。目标：本研究探讨了演化软件分析中复杂工具流水线所面临的有效性威胁，并评估了针对相同研究问题时，工具在数据、研究结果及结论方面的一致性。方法：我们通过轻量级文献综述，选取了来自高水平会议的三项关于协作与协调、软件维护及软件质量的研究，并利用四种独立、系统选择的挖掘工具对这些研究进行形式化复现，以定量和定性方式比较提取的数据、分析结果及结论。结果：我们发现，工具设计与实现中的诸多技术细节在复杂挖掘流水线中不断累积，可能导致提取的基线数据、其衍生数据、后续统计分析结果，以及在特定情况下的结论存在显著差异。结论：用户必须谨慎选择工具并评估其局限性，以恰当方式界定有效性范围。建议复用现有工具。研究人员和工具开发者可通过提供复现包及遵循本方法的比较研究，促进工具的可复用性并帮助减少不确定性。