Beyond Retrieval: A Study of Using LLM Ensembles for Candidate Filtering in Requirements Traceability

by Dominik Fuchß , Stefan Schwedt , Jan Keim , and Tobias Hey

To be published at the 33rd International Requirements Engineering Conference Workshops (REW).

Abstract

[Introduction] Requirements traceability is essential in software development, supporting tasks such as system understanding and change impact analysis. Traceability link recovery (TLR) methods, including those using large language models (LLMs) or retrieval-augmented generation (RAG), often rely on information retrieval (IR) to identify candidate artifact pairs. They are sensitive to hyperparameters (e.g., top-k, similarity thresholds) that require extensive, project-specific tuning.

[Methods] We propose an inter-requirements TLR approach that uses an ensemble of small LLMs (or small language models (SLM)) to incrementally reduce the search space, aiming to replace IR-based candidate selection. We first analyze the sensitivity of IR methods to hyperparameters, then evaluate the ability of small LLMs to filter unrelated requirement pairs, and compare their performance when integrated into a TLR approach. The evaluation includes five projects from the requirements engineering community.

[Results] We find that IR performance heavily depends on project-specific hyperparameter tuning. Furthermore, small LLMs effectively reduce the candidate space with minimal loss of recall. While our LLM-based ensemble approach achieves comparable F2-scores to IR methods, it lags in precision.

[Conclusion] This work provides insights into the capabilities of small LLMs as a filter in inter-requirements TLR. Moreover, it provides insights into the performance of traditional IR techniques for TLR and their dependency on hyperparameters.

Links

• Paper on KITopen
• Replication Package on Zenodo and the corresponding GitHub repository