LLM-Assisted Software Fault Detection and Automated Bug Repair Using Mutation Testing: An Experimental Evaluation of Prompting and Test-Feedback Techniques

doi:10.23940/ijpe.26.11.p3.644654

Abstract

Abstract: The use of Large Language Models (LLMs) in automated software testing and debugging is a growing area of interest. This study is an experimental investigation of the effectiveness of LLM prompting techniques for software fault detection, fault localization, and automated bug repair using mutation testing. A total of 159 lines of code and 33 test cases were evaluated in four Python programs. 32 mutants were created with six mutation operators, and 28 mutants were killed, resulting in an overall mutation score of 87.5%. Three prompting strategies (zero-shot, few-shot, and iterative test-feedback) were compared based on Fault Detection Rate (FDR), Fault Localization Accuracy (FLA), Repair Success Rate (RSR), Precision, Recall, and F1-score. The few-shot prompting and the zero-shot prompting methods showed balanced detection performance with the highest FDR (92.9%) and F1-score (91.2%) respectively. The experimental findings showed that there was a significant difference between the performance of fault detection and fault debugging. The prompting strategies yielded high fault-detection performance, but their effectiveness was significantly reduced for fault localization. The results suggest that the evaluated LLM prompting techniques were more successful in detecting incorrect behavior than in accurately determining the specific code locations that are responsible for the behavior or in creating validated corrective changes.

Key words: large language models, mutation testing, automated bug repair, zero-shot prompting, few-shot prompting, test feedback

Sushil Bhardwaj, Neeraj Kumar Sharma, Tejinder Pal Singh Brar, and Indu Sharma. LLM-Assisted Software Fault Detection and Automated Bug Repair Using Mutation Testing: An Experimental Evaluation of Prompting and Test-Feedback Techniques [J]. Int J Performability Eng, 2026, 22(11): 644-654.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References

[1] Wu Y., Li Z., Zhang J.M., Papadakis M., Harman M., andLiu Y., 2023. Large language models in fault localization.Arxiv Preprint Arxiv:2308.15276.
[2] Schäfer M., Nadi S., Eghbali A., andTip F., 2023. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering,50(1), pp. 85-105.
[3] Zhang Q., Fang C., Sun W., Liu Y., He T., Hao X., andChen Z., 2024. Appt: boosting automated patch correctness prediction via fine-tuning pre-trained models. IEEE Transactions on Software Engineering,50(3), pp. 474-494.
[4] Zhou X., Xu B., Kim K., Han D., Nguyen H.H., Le-Cong T., He J., Le B., andLo D., 2024. Leveraging large language model for automatic patch correctness assessment. IEEE Transactions on Software Engineering,50(11), pp. 2865-2883.
[5] Degiovanni R., andPapadakis M., 2022. µBert: mutation testing using pre-trained language models. In2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 160-169.
[6] Khanfir A., Degiovanni R., Papadakis M., andTraon Y.L., 2023. Efficient mutation testing via pre-trained language models.Arxiv Preprint Arxiv:2301.03543.
[7] Ojdanic M., Garg A., Khanfir A., Degiovanni R., Papadakis M., andLe Traon Y., 2023. Syntactic versus semantic similarity of artificial and real faults in mutation testing studies. IEEE Transactions on Software Engineering,49(7), pp. 3922-3938.
[8] Kang S., Yoon J., andYoo S., 2023. Large language models are few-shot testers: exploring LLM-based general bug reproduction. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 2312-2323.
[9] Chen X., Lin M., Schärli N., andZhou D., 2024. Teaching large language models to self-debug. InInternational Conference on Learning Representations, 2024, pp. 8746-8825.
[10] Yang A.Z., Le Goues C., Martins R., andHellendoorn V., 2024. Large language models for test-free fault localization. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp. 1-12.
[11] Hou X., Zhao Y., Liu Y., Yang Z., Wang K., Li L., Luo X., Lo D., Grundy J., andWang H., 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology,33(8), pp. 1-79.
[12] Wang J., Huang Y., Chen C., Liu Z., Wang S., andWang Q., 2024. Software testing with large language models: survey, landscape, and vision. IEEE Transactions on Software Engineering,50(4), pp. 911-936.
[13] Xia C.S., Wei Y., andZhang L., 2023. Automated program repair in the era of large pre-trained language models. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 1482-1494.
[14] Zhang Q., Fang C., Ma Y., Sun W., andChen Z., 2023. A survey of learning-based automated program repair. ACM Transactions on Software Engineering and Methodology,33(2), pp. 1-69.
[15] Sobania D., Briesch M., Hanna C., andPetke J., 2023. An analysis of the automatic bug fixing performance of ChatGPT. In2023 IEEE/ACM International Workshop on Automated Program Repair (APR), pp. 23-30.
[16] Xia C.S., andZhang L., 2022. Less training, more repairing please: revisiting automated program repair via zero-shot learning. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 959-971.
[17] Xia C.S., andZhang L., 2024. Automated program repair via conversation: fixing 162 out of 337 bugs for $0.42 each using ChatGPT. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 819-831.
[18] Olausson T.X., Inala J.P., Wang C., Gao J., andSolar-Lezama A., 2024. Is self-repair a silver bullet for code generation?. InInternational Conference on Learning Representations, 2024, pp. 36545-36593.
[19] Kang S., An G., andYoo S., 2024. A quantitative and qualitative evaluation of LLM-based explainable fault localization. Proceedings of the ACM on Software Engineering,1(FSE), pp. 1424-1446.
[20] Wang B., Chen M., Deng M., Lin Y., Harman M., Papadakis M., andZhang J.M., 2026. A comprehensive study on large language models for mutation testing.ACM Transactions on Software Engineering and Methodology.
[21] Dakhel A.M., Nikanjam A., Majdinasab V., Khomh F., andDesmarais M.C., 2024. Effective test generation using pre-trained large language models and mutation testing.Information and Software Technology, 171, 107468.