Comparing Fault Detection Efficiencies of Adaptive Random Testing and Greedy Combinatorial Testing for Boolean-Specifications

doi:10.23940/ijpe.21.01.p11.114122

Int J Performability Eng ›› 2021, Vol. 17 ›› Issue (1): 114-122.doi: 10.23940/ijpe.21.01.p11.114122

• Orginal Article • Previous Articles Next Articles

Comparing Fault Detection Efficiencies of Adaptive Random Testing and Greedy Combinatorial Testing for Boolean-Specifications

Ziyuan Wang^a^*, Yanliang Zhang^a, Peng Gao^b, and Shiyong Shuang^c

^aSchool of Computer Science and Technology, Nanjing University of Posts and Telecommunications, Nanjing, 210023, China
^bPLA Strategic Support Force Information Engineering University, Zhengzhou, 450002, China
^cBeijing Jinghang Research Institute of Computing and Communications, Beijing, 100074, China

Submitted on ; Revised on ; Accepted on
Contact: * Corresponding author. E-mail address: wangziyuan@njupt.edu.cn
Supported by:
the Foundation of Nanjing University of Posts and Telecommunications (NY219079)

Abstract

Abstract:

Both random testing and combinatorial testing are input-domain testing techniques. Adaptive random testing, which is an improved version of random testing, selects a test case with more differences from all the existing test cases in each step. Greedy combinatorial testing generates test cases using greedy algorithms to cover more uncovered tuple-combinations of parametric values in each step. To compare fault detection efficiencies of the adaptive random testing technique and greedy combinatorial testing technique, we design an experiment on Boolean specifications that were extracted from the TCAS system. By analyzing fault detection ratios, f-measure values, and APFD values of the two testing techniques, experimental results show that: (1) if the number of test cases is relatively small, fault detection efficiencies of the two techniques are very close though adaptive random testing has a little advantage; (2) for an increase in the number of test cases, the fault detection efficiency of greedy combinatorial testing becomes gradually better.

Key words: input-domain testing, adaptive random testing, greedy combinatorial testing, fault detection efficiency

Ziyuan Wang, Yanliang Zhang, Peng Gao, and Shiyong Shuang. Comparing Fault Detection Efficiencies of Adaptive Random Testing and Greedy Combinatorial Testing for Boolean-Specifications [J]. Int J Performability Eng, 2021, 17(1): 114-122.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References 18.

1.	P. Bourque and R. E. Fairley, “SWEBOK: Guide to the Software Engineering Body of Knowledge (v3.0),” IEEE Computer Society, 2014
2.	T. Y. Chen, T. H. Tse,Y. T. Yu, “Proportional Sampling Strategy: A Compendium and Some Insights,” Journal of Systems and Software, Vol. 58, No. 1, pp. 65-81, 2001
3.	D. M.Cohen and S. R. Dalal, “The AETG System: An Approach to Testing Based on Combinatorial Design,” IEEE Transactions on Software Engineering, Vol. 23, No. 7, pp. 437-444, 1997
4.	R. Huang, W. Sun, Y. Xu, H. Chen, D. Towey,X. Xia, “A Survey on Adaptive Random Testing,”IEEE Transactions on Software Engineering, DOI: 10.1109/TSE.2019.2942921
5.	L. Hu, W. E. Wong, D. R. Kuhn,R. N. Kacker, “How does Combinatorial Testing Perform in the Real World: An Empirical Study,” Empirical Software Engineering, Vol. 25, No. 4, pp. 2661-2693, 2020
6.	C. Nie and H. Leung, “A Survey of Combinatorial Testing,” ACM Computing Surveys (CSUR), Vol. 43, No. 2, pp. 11, 2011
7.	H. Wu, C. Nie, J. Petke, Y. Jia,M. Harman, “An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing,”IEEE Transactions on Software Engineering, DOI: 10.1109/TSE.2018.2852744
8.	N. G. Leveson, M. P. E.Heimdahl, H. Hildreth, and J. D. Reese, “Requirements Specification for Process-Control Systems,” IEEE Transaction Software Engineering, Vol. 20, No. 9, pp. 684-707, 1994
9.	Z. Chen, T. Y. Chen,B. Xu, “A Revisit of Fault Class Hierarchies in General Boolean Specifications,” ACM Transactions on Software Engineering and Methodology (TOSEM), Vol. 20, No. 3, pp. 13, 2011
10.	R. C.Bryce and C. J. Colbourn, “Constructing Interaction Test Suites with Greedy Algorithms,” in Proceedings of 20th IEEE/ACM International Conference on Automated Software Engineering (ASE2005), pp. 440-443, Long Beach, California, USA, November 2005
11.	R. C. Bryce, C. J. Colbourn,M. B. Cohen, “A Framework of Greedy Methods for Constructing Interaction Test Suites,” in Proceedings of 27th International Conference on Software Engineering (ICSE2005), pp. 146-155, St. Louis, Missouri, USA, May 2005
12.	R. C.Bryce and C. J. Colbourn, “One-Test-at-a-Time Heuristic Search for Interaction Test Suites,” in Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation (GECCO2007), pp. 1082-1089, London, England, United Kingdom, July 2007
13.	Z. Wang and F. She, “Combinatorial Test Case Prioritization based on Incremental Combinatorial Coverage Strength,” International Journal of Performability Engineering, Vol. 14, No. 6, pp. 1283-1290, June 2018
14.	T. Y. Chen, F. -C. Kuo, and R. G. Merkel, “On the Statistical Properties of the F-Measure,” in Proceedings of the 4th International Conference on Quality Software (QSIC2004), pp. 146-153, Braunschweig, Germany, September 2004
15.	Z. Wang, C Fang, L Chen,Z. Zhang, “A Revisit of Metrics for Test Case Prioritization Problems,” International Journal of Software Engineering and Knowledge Engineering, Vol. 30, No. 8, pp. 1139-1167, August 2020
16.	E. Weyuker, T. Goradia,A. Singh, “Automatically Generating Test Data from a Boolean Specification,” IEEE Transaction Software Engineering, Vol. 20, No. 5, pp. 353-363, 1994
17.	N. Kobayashi, T. Tsuchiya,T. Kikuno, “Non-Specification-based Approaches to Logic Testing for Software,” Information and Software Technology, Vol. 44, No. 2, pp. 113-121, 2002
18.	R. C.Bryce and C. J. Colbourn, “A Density-based Greedy Algorithm for Higher Strength Covering Arrays,” Software Testing, Verification and Reliability, Vol. 19, No. 1, pp. 37-53, 2009

Comparing Fault Detection Efficiencies of Adaptive Random Testing and Greedy Combinatorial Testing for Boolean-Specifications

PDF

Knowledge

Abstract

Cite this article

share this article

References 18.

Related Articles 1

Recommended 0