Proceedings of the Faculty of Technical Sciences

Information Systems Engineering

Vol. 41 No. 07 (2026): Proceedings of the Faculty of Technical Sciences

Use of Large Language Models in Software Testing

  • David Feješ
DOI:
https://doi.org/10.24867/35OI03Fejes
Submitted
November 2, 2026
Published
2026-07-15

Abstract

The advancement of artificial intelligence has enabled the broad adaptation of large language models across various domains of software development. These models are increasingly used to support automation, analysis, and evaluation of tests, thereby improving the overall efficiency of software testing. This paper explores the role and potential of large language models in modern software testing, with particular emphasis on their advantages, limitations, and factors that affect reliability. Special attention is given to the issues of consistency and stability, which represent major challenges in real-world problems. This finding suggests that large language models currently provide the greatest value as intelligent support tools that complement existing methods, while their full autonomy in testing remains a subject of ongoing research.

References

  1. [1] R. F. de L. Junior, L. F. P. de B. Presta, L. S. Borborema, V. N. da Silva, M. L. de M. Dahia, and A. C. S. e Santos, “A Case Study on Test Case Construction with Large Language Models: Unveiling Practical Insights and Challenges,” Dec. 2023, [Online]. Available: http://arxiv.org/abs/2312.12598
  2. [2] M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, “An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation,” Dec. 2023, [Online]. Available: http://arxiv.org/abs/2302.06527
  3. [3] Y. Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “ChatUniTest: A Framework for LLM-Based Test Generation,” in FSE Companion - Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, Association for Computing Machinery, Inc, Jul. 2024, pp. 572–576. doi: 10.1145/3663529.3663801.
  4. [4] S. Karpurapu et al., “Comprehensive Evaluation and Insights Into the Use of Large Language Models in the Automation of Behavior-Driven Development Acceptance Test Formulation,” IEEE Access, vol. 12, pp. 58715–58721, 2024, doi: 10.1109/ACCESS.2024.3391815.
  5. [5] G. Yi, Z. Chen, Z. Chen, W. E. Wong, and N. Chau, “Exploring the Capability of ChatGPT in Test Generation,” in Proceedings - 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security Companion, QRS-C 2023, Institute of Electrical and Electronics Engineers Inc., 2023, pp. 72–80. doi: 10.1109/QRS-C60940.2023.00013.
  6. [6] A. Lops, F. Narducci, A. Ragone, and M. Trizio, “AgoneTest: Automated creation and assessment of Unit tests leveraging Large Language Models,” in Proceedings - 2024 39th ACM/IEEE International Conference on Automated Software Engineering, ASE 2024, Association for Computing Machinery, Inc, Oct. 2024, pp. 2440–2441. doi: 10.1145/3691620.3695318.
  7. [7] H. Koziolek, V. Ashiwal, S. Bandyopadhyay, and C. K. R, “Automated Control Logic Test Case Generation using Large Language Models,” May 2024, [Online]. Available: http://arxiv.org/abs/2405.01874
  8. [8] A. Guo et al., “SoVAR: Build Generalizable Scenarios from Accident Reports for Autonomous Driving Testing,” in Proceedings - 2024 39th ACM/IEEE International Conference on Automated Software Engineering, ASE 2024, Association for Computing Machinery, Inc, Oct. 2024, pp. 268–280. doi: 10.1145/3691620.3695037.
  9. [9] Z. Wang, K. Liu, G. Li, and Z. Jin, “HITS: High-coverage LLM-based Unit Test Generation via Method Slicing,” Aug. 2024, [Online]. Available: http://arxiv.org/abs/2408.11324
  10. [10] T. Le, T. Tran, D. Cao, V. Le, T. Nguyen, and V. Nguyen, “KAT: Dependency-aware Automated API Testing with Large Language Models,” Jul. 2024, doi: 10.1109/ICST60714.2024.00017.
  11. [11] L. Yang et al., “On the Evaluation of Large Language Models in Unit Test Generation,” in Proceedings - 2024 39th ACM/IEEE International Conference on Automated Software Engineering, ASE 2024, Association for Computing Machinery, Inc, Oct. 2024, pp. 1607–1619. doi: 10.1145/3691620.3695529.
  12. [12] T.-O. Li et al., “Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting,” Sep. 2023, [Online]. Available: http://arxiv.org/abs/2304.11686
  13. [13] S. Kang, J. Yoon, N. Askarbekkyzy, and S. Yoo, “Evaluating Diverse Large Language Models for Automatic and General Bug Reproduction,” Nov. 2023, [Online]. Available: http://arxiv.org/abs/2311.04532
  14. [14] S. Bhatia, T. Gandhi, D. Kumar, and P. Jalote, “Unit Test Generation using Generative AI : A Comparative Performance Analysis of Autogeneration Tools,” Feb. 2024, [Online]. Available: http://arxiv.org/abs/2312.10622
  15. [15] A. M. Dakhel, A. Nikanjam, V. Majdinasab, F. Khomh, and M. C. Desmarais, “Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing,” Aug. 2023, [Online]. Available: http://arxiv.org/abs/2308.16557
  16. [16] C. Lemieux, J. Priya Inala, S. K. Lahiri, and S. Sen, “CODAMOSA: Escaping Coverage Plateaus in Test Generation with Pre-trained Large Language Models.” [Online]. Available: https://github.com/microsoft/codamosa.
  17. [17] K. El Haji, C. Brandt, and A. Zaidman, “Using GitHub Copilot for Test Generation in Python: An Empirical Study,” in Proceedings - 2024 IEEE/ACM International Conference on Automation of Software Test, AST 2024, Association for Computing Machinery, Inc, Apr. 2024, pp. 45–55. doi: 10.1145/3644032.3644443.
  18. [18] S. Kang, J. Yoon, and S. Yoo, “Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction,” Jul. 2023, [Online]. Available: http://arxiv.org/abs/2209.11515
  19. [19] M. Nabeel, D. D. Nimara, and T. Zanouda, “Test Code Generation for Telecom Software Systems using Two-Stage Generative Model,” Apr. 2024, [Online]. Available: http://arxiv.org/abs/2404.09249
  20. [20] J. Hoffmann and D. Frister, “Generating Software Tests for Mobile Applications Using Fine-Tuned Large Language Models,” in Proceedings - 2024 IEEE/ACM International Conference on Automation of Software Test, AST 2024, Association for Computing Machinery, Inc, Apr. 2024, pp. 76–77. doi: 10.1145/3644032.3644454.
  21. [21] N. Alshahwan et al., “Automated Unit Test Improvement using Large Language Models at Meta,” in FSE Companion - Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, Association for Computing Machinery, Inc, Jul. 2024, pp. 185–196. doi: 10.1145/3663529.3663839.
  22. [22] A. Deljouyi, “Understandable Test Generation Through Capture/Replay and LLMs,” in Proceedings - International Conference on Software Engineering, IEEE Computer Society, Apr. 2024, pp. 261–263. doi: 10.1145/3639478.3639789.
  23. [23] L. Plein, W. C. Ouédraogo, J. Klein, and T. F. Bissyandé, “Automatic Generation of Test Cases based on Bug Reports: A Feasibility Study with Large Language Models,” in Proceedings - International Conference on Software Engineering, IEEE Computer Society, Apr. 2024, pp. 360–361. doi: 10.1145/3639478.3643119.