Large Language Models (LLMs) demonstrate exceptional capabilities across general domains, yet their application to specialized sectors such as mortgage finance requires domain-specific knowledge augmentation while preserving instruction-following fidelity. We present MortgageLLM, a novel domain-specific large language model that addresses this dual challenge. It is developed using a dual-track specialization framework from a single base model (LLaMA-3.1-8B). We opted for this dual-expert approach as a single multi-task model suffers from performance trade-offs, where optimizing for structured tasks (via SFT) degrades conversational fidelity (via DPO). Our dual-track method solves this by creating two specialists, allowing each to be optimally trained for its distinct capability. Our approach applies the instruction residual technique to restore instruction-following capabilities post-domain adaptation without supervised fine-tuning. We contribute: (1) application of this residual technique to the highly specialized mortgage finance domain; (2) a dual-expert architecture combining a conversational Q&A model and a structured task model for classification and summarization; and (3) an intelligent task routing mechanism using few-shot classification performed by one of the expert models itself. We validate our approach on domain-specific benchmarks, where our final model (MLM v2) significantly outperforms the base LLaMA-3.1-8B-Instruct, achieving an LLM-as-a-Judge summarization score of 4.58 (vs. 3.99), a Q&A score of 4.09 (vs. 4.0), and a classification score of 2.6 (vs. 1.2). On semantic similarity, our model achieved a BERTScore of 0.77 for summarization (vs. 0.74), 0.68 for Q&A (vs. 0.58), and 0.75 for classification (vs. 0.73), substantially outperforming baseline approaches.

翻译：大语言模型（LLMs）在通用领域展现出卓越能力，但在抵押贷款金融等专业领域的应用，需要在保持指令遵循忠实度的同时增强领域特定知识。本文提出MortgageLLM，一种新颖的领域专用大语言模型，以应对这一双重挑战。该模型基于单一基础模型（LLaMA-3.1-8B），通过双轨专业化框架开发。我们采用双专家方法，因为单一多任务模型存在性能权衡问题——针对结构化任务（通过监督微调SFT）优化会损害对话忠实度（通过直接偏好优化DPO）。我们的双轨方法通过创建两个专家模型解决了这一问题，使每个模型能针对其特定能力进行最优训练。我们应用指令残差技术，在领域自适应后无需监督微调即可恢复指令遵循能力。本文贡献包括：（1）将残差技术应用于高度专业化的抵押贷款金融领域；（2）结合对话问答模型与结构化任务模型（用于分类与摘要）的双专家架构；（3）由专家模型自身执行少样本分类的智能任务路由机制。我们在领域特定基准上验证了该方法，最终模型（MLM v2）显著优于基础模型LLaMA-3.1-8B-Instruct：LLM-as-a-Judge摘要得分达4.58（对比3.99），问答得分4.09（对比4.0），分类得分2.6（对比1.2）；在语义相似度方面，摘要任务BERTScore达0.77（对比0.74），问答任务0.68（对比0.58），分类任务0.75（对比0.73），全面超越基线方法。