Small fine-tuned model asks for clarification in RAG systems

Findings

The researchers studied query clarification in retrieval-augmented generation, or RAG, systems. They considered multi-turn conversations in which a query could be poorly formed, ambiguous in light of the earlier dialogue, or open to several relevant interpretations when compared with retrieved documents. The task was to recognize when clarification was needed and generate an appropriate follow-up request.

They introduced ClariEdge, a dataset containing approximately 3,900 examples. It includes ambiguous queries and negative examples that do not require clarification. According to the preprint, a lightweight fine-tuned LoRA achieved 96.6% classification accuracy in deciding whether clarification was required.

The authors report that the LoRA outperformed prompted baselines from many larger models in both classification and the quality of generated clarification requests. They also report that difficult negative examples helped distinguish genuinely ambiguous queries from queries that appeared ambiguous but could be resolved from context. The same LoRA reportedly performed strongly in scenarios before and after document retrieval.

Why it matters

The researchers propose using the model as a check at two points in a RAG pipeline: before retrieval and after retrieval but before answer generation. If the reported results hold in other settings, this approach could help systems identify when they should seek more information rather than choose among multiple interpretations. However, the abstract reports benchmark performance, not evidence of improved outcomes for people using deployed systems.

What remains uncertain

This work is an arXiv preprint and has not been peer reviewed. Although the abstract identifies ClariEdge and its approximate size, it does not explain the evaluation splits or whether additional evaluation datasets were used. It also does not identify the underlying model or individual larger-model baselines, define how clarification quality was measured, report uncertainty intervals or the size of the advantage over baselines, or provide computational-cost and real-world user-study results. The fairness of the comparisons and generalisation beyond the reported evaluation therefore cannot be assessed from the abstract.