Testing Machine Learning Classifiers based on Compositional Metamorphic Relations

doi:10.23940/ijpe.20.01.p8.6777

Abstract

Abstract:

With the widespread application of intelligent software, more rigorous requirements are placed on the security and reliability of intelligent software programs. The major challenge is that the traditional test approaches cannot be easily adapted to the testing of intelligent software since the test oracle is not available and testing intelligent software needs to focus on training sets. The classifier is a typical intelligent software which has an uncertain output. As a result, the accuracy rate cannot be used to judge whether the classifier is defective. Therefore, this paper proposes an approach for testing classifiers based on compositional metamorphic relations. Initially, it was recommended to construct composite metamorphic relations to generate derivative test cases from the original test cases; followed by training classifier to predict the classification of the test data set; then checks their consistency between the original test cases and the derivative test cases against compositional metamorphic relations, and the detected violations are reported as bugs. The experiment is carried on the ID3 decision tree and compares the mutant detection capability with the on one-dimensional metamorphic testing. From the results received from the experiment conducted in this research shows that the proposed approach improves 16.7% of mutant kill rates.

Key words: intelligent software test, metamorphic test, mutation test, compositional metamorphic relations, classifiers, decision tree

Minghua Jia, Xiaodong Wang, Yue Xu, Zhanqi Cui, and Ruilin Xie. Testing Machine Learning Classifiers based on Compositional Metamorphic Relations [J]. Int J Performability Eng, 2020, 16(1): 67-77.

Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks

Figures/Tables 14

Figure 1.

Figure 2.

Figure 3.

Algorithm 2. Against metamorphic relations
Input: Set: $PR α$ and $PR β$ //Original use case prediction results and derived use case prediction results Output: Set: bugList //The result of a set of metamorphic relation consistency verification
1:for each $D i$ in $PR β$ do 2:// $D i$ represents different data sets of the same metamorphic relations 3: for i=0; i<=L; i++ do 4: //L represents the number of rows of $PR α$ and $PR β$ 5: if $PR αi$ and $PR βi$ dissatisfy the metamorphic relations then 6: bugList.add( $L I$ ) 7: end if 8: end for 9:end for
10:return bugList

Table 1

Table 2

Table 3

Table 4

Table 5

Figure 4.

References 19.

1.	Z. Markel and M. Bilzor, “Building a Machine Learning Classifier for Malware Detection,” inProceedings of 2014 Second Workshop on Anti-malware Testing Research (WATeR), pp. 1-4, Canterbury, 2014
2.	M. Lei, L.Yang, Z. J. Jun, W. Y. Dong, J. X. Felix, Z. F. Yuan, et al.,“DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems,” inProceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE), pp. 120-131, Montpellier, France, September 2018
3.	X. X. Yuan, W. K. H. Joshua, M. Christian, K. Gail, X. B. Wen, and C. Y. Tsong, “Testing and Validating Machine Learning Classifiers by Metamorphic Testing,” Journal of Systems and Software (online since December 7, 2010))(DOI 10.1016/S0164121210003213)
4.	N. Shin and B. N. Hai, “Dataset Coverage for Testing Machine Learning Computer Programs,” inProceedings of 2016 23rd Asia-Pacific Software Engineering Conference (APSEC), pp. 297-304, Hamilton, 2016
5.	S. Al-Azani and J. Hassine,“Validation of Machine Learning Classifiers using Metamorphic Testing and Feature Selection Techniques,” inProceedings of International Workshop on Multi-disciplinary Trends in Artificial Intelligence (MIWAI), pp. 77-91, Springer, Cham, 2017
6.	A. Ozcift and A. Gulten,“Classifier Ensemble Construction with Rotation Forest to Improve Medical Diagnosis Performance of Machine Learning Algorithms,” Computer Methods and Programs in Biomedicine(online since March 2011)2011.03.018)
7.	L. R. Chao, L. X. Ming, H. Song,H. Z. Wei,“Metamorphic Relation Construction Method based on Compositional Function,” Command Information System and Technology(online since February 2013)2013.01.016)
8.	D. G. Wei, X. B. Wen, C. Lin, N. C. Hai,W. L. Lu,“Cases Studies on Testing with Compositional Metamorphic Relations,” Journal of Southeast University(online since December 2008)2008.04.009)
9.	L. Huai, L. Xuan,C. T. Yueh, “A New Method for Constructing Metamorphic Relations,” inProceedings of 2012 12th International Conference on Quality Software (QSIC), pp. 59-68, Xi'an, China, October 2012
10.	P. Harrington.Dection Tree, In Machine Learning in Action, People Post Press, pp. 32-52, 2013
11.	Z. Jie, W. Z. Yi, Z. L. Ming,H. Dan,“Predictive Mutation Testing,” IEEE Transactions on Software Engineering (online since July2016) (DOI 10.1145/2931037.2931038)
12.	Z. Jing, H. X. Gang,Z. Bin,“Test Approach for the Program of Clusters based on Metamorphic Relations,” Journal of Electronic Measurement and Instrument(online since August 2011)2011.00688)
13.	Benoit Julien, “UCI machine learning repository,”
14.	M. Bellare, A. Boldyreva,A. Palacio, “An Uninstantiable Random-Oracle-Model Scheme for a Hybrid-Encryption Problem,” inProceedings of International Conference on the Theory and Applications of Cryptographic Techniques (EUROCRYPT), pp. 171-188, Heidelberg, Germany, 2014
15.	J. Vondrák, “Optimal Approximation for the Submodular Welfare Problem in the Value Oracle Model,” inProceedings of the 40th Annual ACM Symposium on Theory of Computing (STOC), pp. 67-74, British, Canada, May 2008
16.	A. Ozcift and A. Gulten, “Classifier Ensemble Construction with Rotation Forest to Improve Medical Diagnosis Performance of Machine Learning Algorithms,” Computer Methods and Programs in Biomedicine(online since March 2011)2011.03.018)
17.	A. Gotlieb and B. Botella, “Automated Metamorphic Testing,” inProceedings of 26th Annual International Computer Software and Applications Conference (COMPSAC), pp. 34-40, December 2003
18.	C. T. Ysong, F. J. Qiang,T. H. Tse, “Metamorphic Testing of Programs on Partial Differential Equations: a Case Study,” inProceedings of 26th Annual International Computer Software and Applications (COMPSAC), pp. 327-333, February 2002
19.	W. Wei and Z. K. Rong, “Researches on Basic Citerion and Strategy of Constructing Metamorphic Relations,” Computer Science, Vol. 39, No. 1, pp. 115-119, January 2012

Operator	Description	Mutants	Number
AOR	Arithmetic Operator Replacement	μ1, μ2	32
CRP	Constant Replacement	μ3, μ4	92
ASR	Short-Cut Assignment Operator Replacement	μ5, μ6	16
SMD	Statement Deletion	μ7, μ8	49
OIL	One Iteration Loop	μ9, μ10	7
RIL	Reverse Iteration Loop	μ11, μ12	7
ZIL	Zero Iteration Loop	μ13, μ14	7

Attribute name	Description	Attribute value
Age	Age of the patient	Young, Pre, Pesbyopic
Prescription	Spectacle prescription	Myope, Hyper
Astigmatic	Astigmatic	Yes, No
Tear rate	Tear production rate	Reduced, Normal
type	Type of glasses	Soft, Hard, No lenses

MR	Content	Specific operation
MR-1	Addition of uninformative attributes.	Add a column of irrelevant items. The data in this column is an unrelated string. This column should not affect the result.
MR-2	Permutation of the attribute.	Exchange any two columns of data, the training results should be unchanged.
MR-3	Additional training sample.	Copy a column, the training results should be unchanged.
MR-4	Permutation of class labels.	Replace the class label in the result column, and the training result should correspond to the transformation.
MR-5	Removal of classes.	Suppose that the classification result of the test sample ts is li, delete all samples of a certain class of the training set that is not li, and the classification result of ts is still li, and remains unchanged.

MR	Number	μ1	μ2	μ5	μ6	μ7	μ8	μ9	μ10	μ11	μ12	μ13	μ14	KR
MR-1	30	0	0	0	0	0	0	0	0	0	0	0	0	0%
MR-2	30	25	23	25	25	6	20	13	0	6	6	25	25	79%
MR-3	24	0	0	0	0	0	0	0	0	0	0	0	0	0%
MR-4	30	25	25	25	25	0	10	5	10	0	0	25	25	71%
MR-5	6	3	3	3	3	0	0	4	1	0	0	3	3	57%
Total	-	3	3	3	3	1	2	3	2	1	1	3	3	86%

MR	Number	μ1	μ2	μ3	μ4	μ5	μ6	μ7	μ8	μ9	μ10	μ11	μ12	μ13	μ14	KR
MR-1.1	150	0	0	28	30	0	0	30	28	12	26	30	30	0	0	57%
MR-1.2	150	116	98	0	0	116	116	24	94	84	0	24	24	116	116	79%
MR-1.3	150	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0%
MR-1.4	150	125	125	0	0	125	125	0	50	25	50	0	0	125	125	64%
MR-1.5	30	10	11	20	20	10	10	20	16	17	17	20	20	10	10	100%
MR-2.2	150	95	88	0	0	95	95	20	76	92	0	20	20	95	95	79%
MR-2.3	120	100	92	0	0	100	100	24	80	52	0	24	24	100	100	79%
MR-2.4	150	130	135	0	0	130	130	30	100	70	50	30	30	130	130	86%
MR-2.5	30	5	5	25	25	5	5	25	20	20	20	25	25	5	5	100%
MR-3.3	120	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0%
MR-3.4	120	100	100	0	0	100	100	0	40	20	40	0	0	100	100	64%
MR-3.5	24	9	10	15	15	9	9	15	12	13	13	15	15	9	9	100%
MR-4.4	150	116	116	117	117	116	116	117	120	116	117	117	117	116	116	100%
MR-4.5	6	4	4	5	5	4	4	5	4	5	5	5	5	4	4	100%
MR-5.5	18	8	8	2	3	8	8	3	1	7	11	11	3	8	8	100%
Total	-	12	12	7	7	12	12	11	13	13	10	11	11	12	12	100%