Abstract: The problem of fault detection capability of combinatorial testing has drawn a lot of attention. People conducted many experiments on different subjects to compare fault detection capabilities of combinatorial testing and random testing. However, previous confusing results can hardly answer the question of whether combinatorial test suite detects more faults than random test suite. To answer this question more trustfully, we conducted an experiment on general-form Boolean-specifications. In our experiment, fault detection frequencies and fault detection ratios in combinatorial testing, where test suites are generated by some classic combinatorial test generation algorithms, are collected by the repeated running of combinatorial testing. Moreover, fault detection probabilities in random testing are obtained by theoretical analysis. By comparing fault detection frequencies, ratios, and probabilities, our experimental results suggest that combinatorial testing has a little advantage of fault detection capability over random testing.

Key words: combinatorial testing, random testing, fault detection probability, fault detection frequency, fault detection ratio