Combinatorial Test Case Prioritization based on Incremental Combinatorial Coverage Strength

doi:10.23940/ijpe.18.06.p19.12831290

Abstract

Abstract:

A combinatorial test case prioritization technique based on incremental combinatorial coverage strength is proposed in this paper. Such an efficient technique prioritizes test cases in an existing high-strength combinatorial test suite, to form an ordered test case sequence. The output prioritized combinatorial test suite could cover combinations of parametric values, where their strengths are mixed from 2 or more, quicker than non-prioritized combinatorial test suite. Theoretical analysis shows that the proposed technique consumes less cost than existing combinatorial test case prioritization technique for the same type of scenarios. Experimental results also show that compared to initial combinatorial test suites without prioritization, prioritized combinatorial test suites yield a higher speed of covering combinations of parametric values. Compared to combinatorial test suites generated by an existing incremental adaptive combinatorial testing strategy, prioritized combinatorial test suites 1) require less test cases to achieve final combinatorial coverage, and 2) yield a higher speed of covering high-strength combinations of parametric values.

Submitted on March 16, 2018; Revised on April 26, 2018; Accepted on May 27, 2018
References: 24

Ziyuan Wang Feiyan She. Combinatorial Test Case Prioritization based on Incremental Combinatorial Coverage Strength [J]. Int J Performability Eng, 2018, 14(6): 1283-1290.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

References 0

	R. C. Bryce, C. J. Colbourn, “Test Prioritization for Pairwise Interaction Coverage,” In Proceedings of 1^st International Workshop on Advances in Model-Based Testing, pp. 1-7, St Louis, Missouri, USA, May 15-21, 2005.
	R. C. Bryce, C. J. Colbourn. A Density-Based Greedy Algorithm for Higher Strength Covering Arrays. Software Testing, Verification and Reliability, 2009, 19(1): 37-53.
	R. C. Bryce, A. M. Memon. “Test Suite Prioritization by Interaction Coverage”. In Proceedings of Workshop on Domain-Specific Approaches to Software Test Automation (DoSTA2007), Dubrovnik, Croatia, September 4, 2007.
	R. C. Bryce, S. Sampath, A. M. Memon. “Developing a Single Model and Test Prioritization Strategies for Event-Driven Software”. IEEE Transactions on Software Engineering, 2011, 37(1): 48-64.
	Xiang Chen, Qing Gu, Xin Zhang, Daoxu Chen. Building Prioritized Pairwise Interaction Test Suites with Ant Colony. In Proceedings of the 9th International Conference on Quality Software (QSIC2009), August 24-25, 2009: 347-352.
	C. J. Colbourn, S, S. Martirosyan, T. V. Trung, R. A. Walker II. Roux-Type Constructions for Covering Arrays of Strengths Three and Four. Designs, Codes and Cryptography, 2006, 41(1): 33-57.
	S. Elbaum, A. Malishevsky, G. Rothermel, “Prioritizing Test Cases for Regression Testing,” in Proceedings of International Symposium on Software Testing and Analysis (ISSTA2000): 102-112.
	S. Elbaum, A. Malishevsky, G. Rothermel. Incorporating Varying Test Costs and Fault Severities into Test Case Prioritization. In Proceedings of the 23rd International Conference on Software Engineering (ICSE2001): 329-338.
	S. Fouche, M. B. Cohen, A. Poter, “Towards Incremental Adaptive Covering Arrays,” in Proceedings of 6^th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering (ESEC/FSE2007), pp. 557-560, Dubrovnik, Croatia, September 3-7, 2007.
	R. Huang, J. Chen, Z. Li, R. Wang, Y Lu. Adaptive random prioritization for interaction test suites. In: Proceedings of the 29th Symposium on Applied Computing (SAC’14): 1058–1063.
	R. Huang, J. Chen, D. Towey, A. T. S. Chan, Y. Lu. “Aggregate-strength interaction test suite prioritization”. Journal of Systems & Software, 2015, 99(C):36-51.
	Rubing Huang, Xiaodong Xie, Dave Towey, Tsong Yueh Chen, Yansheng Lu, Jinfu Chen. “Prioritization of Combinatorial Test Cases by Incremental Interaction Coverage”. International Journal of Software Engineering and Knowledge Engineering, 2013.
	D. Jeffrey, N. Gupta. Test Case Prioritization Using Relevant Slices. In Proceedings of the 29th International Computer Software and Applications Conference (COMPSAC2006), Chicago, USA. pp. 411-420.
	Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, J. Lawrence. IPOG IPOG-D: Efficient Test Generation for Multi-Way Combinatorial Testing. Software Testing, Verification and Reliability, 2008, 18(3): 125-148.
	R. Mandl. Orthogonal Latin Squares: An Application of Experimental Design to Compiler Testing. In Communications of the ACM, 1985, 28(10): 1054-1058.
	Changhai Nie, Hareton Leung. “A survey of combinatorial testing”. ACM Computing Surveys (CSUR), 2011, 43(2):1-29.x
	G. Rothermel, R. H. Untch, C. Y. Chu, M. J. Harrold, “Prioritizing Test Cases for Regression Testing,” IEEE Transactions on Software Engineering, 2001, 27(10): 929-948.
	A. Srivastava, J. Thiagrajan. Effectively Prioritizing Tests in Development Environment. In Proceedings of the ACM International Symposium on Software Testing and Analysis (ISSTA2002), Roma, Italy. pp. 97-106.
	K. R. Walcott, M. L. Soffa, G. M. Kapfhammer, R. S. Roos. Time-Aware Test Suite Prioritization. In Proceedings of the ACM International Symposium on Software Testing and Analysis (ISSTA2006), Portland, Maine, USA. pp. 1-12.
	Zi-Yuan Wang. “Test Case Generation and Prioritization for Combinatorial Testing”. A Dissertation for Ph.D., Southeast University, 2009.
	Ziyuan Wang, Baowen Xu, Changhai Nie. “Greedy Heuristic Algorithms to Generate Variable Strength Combinatorial Test Suite”. In Proceedings of the 8th International Conference on Quality Software (QSIC2008), Oxford, UK, August 12-13, 2008: 155-160.
	Z. Wang, L. Chen, B. Xu, and Y. Huang. “Cost-Cognizant Combinatorial Test Case Prioritization”. International Journal of Software Engineering and Knowledge Engineering, 2011, 21(06):829-854.
	W. E. Wong, J. R. Horgan, S. London, H. Agrawal. A Study of Effective Regression Testing in Practice. In Proceedings of the 8th IEEE International Symposium on Software Reliability Engineering (ISSRE1997): 264-274.
	X. Zhang, B. Qu, “An Improved Metric for Test Case Prioritization,” In Proceedings of Web Information Systems and Applications Conference (WISA2011): 125-130.