Int J Performability Eng ›› 2019, Vol. 15 ›› Issue (11): 2952-2961.doi: 10.23940/ijpe.19.11.p14.29522961

Previous Articles     Next Articles

Fault Detection Capabilities of Combinatorial Testing and Random Testing for Boolean-Specifications

Ziyuan Wanga,b,c,*, Yang Lia, Xueqing Gua, Xiaojia Zhenga, and Min Yua   

  1. aSchool of Computer Science and Technology, Nanjing University of Posts and Telecommunications, Nanjing, 210023, China;
    bSoftware Testing Engineering Laboratory of Jiangsu Province, Nanjing, 211169, China;
    cTongda College, Nanjing University of Posts and Telecommunications, Yangzhou, 225127, China
  • Submitted on ; Revised on ; Accepted on
  • Contact: * E-mail address:

Abstract: The problem of fault detection capability of combinatorial testing has drawn a lot of attention. People conducted many experiments on different subjects to compare fault detection capabilities of combinatorial testing and random testing. However, previous confusing results can hardly answer the question of whether combinatorial test suite detects more faults than random test suite. To answer this question more trustfully, we conducted an experiment on general-form Boolean-specifications. In our experiment, fault detection frequencies and fault detection ratios in combinatorial testing, where test suites are generated by some classic combinatorial test generation algorithms, are collected by the repeated running of combinatorial testing. Moreover, fault detection probabilities in random testing are obtained by theoretical analysis. By comparing fault detection frequencies, ratios, and probabilities, our experimental results suggest that combinatorial testing has a little advantage of fault detection capability over random testing.

Key words: combinatorial testing, random testing, fault detection probability, fault detection frequency, fault detection ratio