Capital One at EMNLP 2025: Trust and efficiency in AI

Capital One is excited to participate in the upcoming 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025) in Suzhou, China, taking place Nov. 4-9, 2025. This year, our keynote talk at the Workshop on Financial Technology and Natural Language Processing, as well as our accepted papers, demonstrate how we’re applying research and deep technical innovation to foundational problems in AI safety, model deployment and real-world system reliability as our researchers work to advance the state of the art in Natural Language Processing (NLP).

Dr. Sambit Sahu, VP, AI Foundations, will be delivering a keynote on Orchestrating LLMs for Complex Financial Reasoning with Multi-Agentic Workflow at the The 10th Workshop on Financial Technology and Natural Language Processing. This talk highlights the forefront of AI/ML in the financial services industry and introduces the Multi-Agent Conversational Assistant Workflow (MACAW), an LLM-based framework we developed to build conversational assistants capable of both answering complex questions and executing actions on behalf of users. MACAW leverages self-reflection, planning and precise API generation to meet user needs. It helps enhance customer interaction, moving beyond simple dialogue handling to delivering dynamic, API-grounded, business-logic-aware solutions that continuously learn and adapt.

Additionally, below are the eight accepted papers authored by Capital One researchers and our academic collaborators at EMNLP 2025.

Our Enterprise AI teams are dedicated to solving critical, industry-scale challenges in building safe and efficient language systems. The following papers, led by Capital One researchers, focus on improving the trustworthiness and deployment of Large Language Models (LLMs).

GRAID: Synthetic Data Generation With Geometric Constraints and Multi-Agentic Reflection for Harmful Content Detection

Addressing the problem of data scarcity in harmful text classification for guardrailing applications, this paper introduces GRAID (Geometric and Reflective AI-Driven Data Augmentation), a novel framework that leverages various models and LLM agents for dataset augmentation. This work is first-authored by Capital One researchers Melissa Kazemi Rad and Alberto Purpura. GRAID operates in two stages:

1. Geometrically Controlled Generation: Uses a constrained LLM fine-tuned to reliably cover the input space and ensure that new generations are effectively steered toward the desired semantic regions.

2. Multi-Agentic Reflection: Employs a reflective agentic process that promotes stylistic and linguistic diversity, uncovers difficult edge cases in harmful content and automatically validates and recalibrates generated data to capture variability while preserving categories.

This combination enables both reliable coverage and nuanced exploration of the problem space. Experiments on two benchmark datasets and two downstream fine-tuned guardrail models demonstrate that augmenting the classification dataset with GRAID leads to significant improvements in downstream guardrail model performance, achieving an average F1 score increase of 12% compared with baselines. Experimental analyses also demonstrate the effectiveness of GRAID in diversifying new generations in unexplored and non-overlapping embedding regions compared with the anchor dataset.

Capital One Authors: Melissa Kazemi Rad, Alberto Purpura, Himanshu Kumar, Emily Chen, Mohammad Sorower

 

A Comparison of Independent and Joint Fine-Tuning Strategies for Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) relies on two LLMs: an embedding model for context retrieval and a generator model for answer generation. Both models can be fine-tuned to boost performance on new tasks, but multiple fine-tuning strategies with different costs and benefits exist. This research, first-authored by Capital One researcher Neal Lawton (originally a 2024 ARIP Intern), evaluates and compares three RAG fine-tuning strategies: independent, joint and two-phase fine-tuning. The findings show that all strategies achieve roughly equal improvement in generation quality, despite having significantly different computational costs. The paper concludes that the optimal fine-tuning strategy depends critically on whether the training dataset includes context labels and whether an exhaustive grid search over learning rates is required.

Capital One Authors: Neal Lawton, Anoop Kumar, Alfy Samuel, Daben Liu

 

Harmonizing Diverse Models: A Layer-Wise Merging Strategy for Consistent Generation

A major hurdle in deploying RAG systems is the LLM’s tendency to generate inconsistent outputs for semantically equivalent inputs, a problem exacerbated by limited consistency-focused data. This paper proposes a new approach that combines systematic synthetic data generation and triplet loss for better embeddings, anchored by a novel layer-wise model-merging approach. By using consistency-aware weights derived from intermediate layer activations, the method effectively integrates knowledge from specialized models. Experimental results show that the merged model significantly enhances output consistency, achieving approximately 47.5% improvement in response similarity over the baseline, offering a practical solution for increasing the reliability of an industrial RAG system.

Capital One Authors: Xujun Peng, Anoop Kumar, Jingyu Wu, Parker Glenn, Daben Liu

Our research efforts extend through strong collaborations with the broader AI community and the impactful work of our technologists before and during their time at Capital One.

TruthTorchLM: A Comprehensive Package for Predicting Truthfulness in LLM Outputs

Generative LLMs inevitably produce untruthful responses, making accurate truthfulness prediction critical in high-impact settings. This paper introduces TruthTorchLM, an open-source Python library featuring over 30 truthfulness prediction methods, referred to as Truth Methods. Unlike existing toolkits limited to uncertainty or document-grounded verification, TruthTorchLM offers a broad and extensible collection of techniques spanning diverse trade-offs in computational cost, access level and supervision type. This research was funded through the USC‒Capital One Center for Responsible AI and Decision Making in Finance (CREDIF), a joint research center dedicated to advancing AI in finance. The work was first-authored by USC PhD students, 2025 Capital One Fellow Duygu Nur Yaldiz and 2025 ARIP Intern Yavuz Faruk Bakman. TruthTorchLM is compatible with Hugging Face and LiteLLM, offering a unified interface for generation, evaluation and calibration.

Capital One Authors: Zhiqi Huang, Anoop Kumar, Alfy Samuel, Daben Liu

 

seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs

This paper introduces seqBench, a parameterized benchmark designed to probe the limits of sequential reasoning in LLMs through precise, multidimensional control over complexity. seqBench allows systematic variation of logical depth, the number of backtracking steps required and the noise ratio in the environment. Evaluations on state-of-the-art LLMs reveal a universal failure pattern: Accuracy collapses exponentially beyond a model-specific logical depth. Co-authored by current Capital One researcher Mo Vazifeh during his time at MIT, seqBench’s fine-grained control illuminates universal scaling laws and statistical limits, underscoring key limitations in LLMs’ commonsense reasoning capabilities despite minimal search complexity. The public Hugging Face release of seqBench datasets is intended to spur deeper scientific inquiry into LLM reasoning.

Capital One Authors: Mo Vazifeh

Capital One research will also be featured at two EMNLP workshops, focusing on trustworthy and adaptable AI deployment.

Confidence-Based Response Abstinence: Improving LLM Trustworthiness via Activation-Based Uncertainty Estimation

(Accepted to the Second Workshop on Uncertainty-Aware NLP)

Confidence estimation is crucial for RAG systems in high-impact domains, where abstaining from an answer is better than providing an incorrect one. This research, first-authored by Capital One researchers Zhiqi Huang and Vivek Datla, presents a method for confidence estimation that extends prior uncertainty quantification by leveraging raw feed-forward network (FFN) activations as auto-regressive signals, avoiding the information loss inherent in token logits. In applied settings, this method outperforms strong baselines and maintains high accuracy under strict latency constraints. The results demonstrate that activation-based confidence modeling offers a scalable, architecture-aware path toward trustworthy RAG deployment.

Capital One Authors: Zhiqi Huang, Vivek Datla, Chenyang Zhu, Alfy Samuel, Daben Liu, Anoop Kumar, Ritesh Soni

 

Readability Reconsidered: A Cross-Dataset Analysis of Reference-Free Metrics

(Accepted to the Fourth Workshop on Text Simplification, Accessibility and Readability (TSAR))

Automatic readability assessment plays a key role in ensuring effective communication in customer support domains. This paper, first-authored by 2025 ARIP Intern Catarina Garcia Belem, investigates the factors shaping human perceptions of readability. It then evaluates a diverse set of metrics across many datasets, showing why certain metrics fall short of being reliable predictors of text accessibility. 

Capital One Author: Parker Glenn

 

Language Surgery in Multilingual Large Language Models

(Accepted to the 5th Multilingual Representation Learning Workshop)

This paper, a result of a collaboration across multiple universities and companies, investigates the naturally emerging representation alignment in LLMs, particularly in the middle layers, and its implications for disentangling language-specific and language-agnostic information. Building on these findings, the paper proposes Inference-Time Language Control (ITLC), a novel method that leverages latent injection to enable precise cross-lingual language control and mitigate language confusion in LLMs. Experiments highlight ITLC’s strong control capabilities while preserving semantic integrity, demonstrating effectiveness in alleviating the cross-lingual language confusion problem that leads to inconsistent language generation. Capital One Applied Researcher Genta Winata was one of the authors on this project.

Capital One Author: Genta Winata

We encourage you to attend the presentations and workshops featuring Capital One's research at EMNLP 2025. This is an excellent opportunity to gain deeper insights into the work our teams are doing to embed AI throughout our business and to solve challenging problems.

We look forward to a fantastic EMNLP 2025!