Int J Performability Eng ›› 2020, Vol. 16 ›› Issue (1): 67-77.doi: 10.23940/ijpe.20.01.p8.6777
• Orginal Article • Previous Articles Next Articles
Minghua Jia, Xiaodong Wang, Yue Xu, Zhanqi Cui*(), and Ruilin Xie
Submitted on
;
Revised on
;
Accepted on
Contact:
Zhanqi Cui
E-mail:czq@bistu.edu.cn
Supported by:
Minghua Jia, Xiaodong Wang, Yue Xu, Zhanqi Cui, and Ruilin Xie. Testing Machine Learning Classifiers based on Compositional Metamorphic Relations [J]. Int J Performability Eng, 2020, 16(1): 67-77.
Add to citation manager EndNote|Reference Manager|ProCite|BibTeX|RefWorks
"
Algorithm 1. Composite program based on compositional metamorphic relations |
---|
Input: Set: MR // n valid one-dimensional metamorphic relations Const: K // K represents the k- dimensional metamorphic relations Output: |
1:for i=0; i<=K; i++ do 2: Select a metamorphic relation from Set MR 3: MR[i]= Selected metamorphic; 4:end for 5:composite (Object []MR, int k, int m) 6:// generate all permutations of MR[k:m] 7:if K==m then // only one metamorphic relation 8: for int i=0; i<=m; i++ do 9: System.out.print(MR[i]); 10:else // recursively produces composite metamorphic relations 11: for int i=k; i<=m; i++ do 12: MyMath.swap(MR,k,i) 13: composite(MR, k, i) 14: MyMath.swap(MR,k,i) 15: end for 16:end if 17:return MR[1:k] |
"
Algorithm 2. Against metamorphic relations |
---|
Input: Set: //Original use case prediction results and derived use case prediction results Output: Set: bugList //The result of a set of metamorphic relation consistency verification |
1:for each 2:// 3: for i=0; i<=L; i++ do 4: //L represents the number of rows of 5: if 6: bugList.add( 7: end if 8: end for 9:end for |
10:return bugList |
Table 1
Mutants of the ID3 decision tree classifier program"
Operator | Description | Mutants | Number |
---|---|---|---|
AOR | Arithmetic Operator Replacement | μ1, μ2 | 32 |
CRP | Constant Replacement | μ3, μ4 | 92 |
ASR | Short-Cut Assignment Operator Replacement | μ5, μ6 | 16 |
SMD | Statement Deletion | μ7, μ8 | 49 |
OIL | One Iteration Loop | μ9, μ10 | 7 |
RIL | Reverse Iteration Loop | μ11, μ12 | 7 |
ZIL | Zero Iteration Loop | μ13, μ14 | 7 |
Table 2
Attributes of the contact lenses dataset"
Attribute name | Description | Attribute value |
---|---|---|
Age | Age of the patient | Young, Pre, Pesbyopic |
Prescription | Spectacle prescription | Myope, Hyper |
Astigmatic | Astigmatic | Yes, No |
Tear rate | Tear production rate | Reduced, Normal |
type | Type of glasses | Soft, Hard, No lenses |
Table 3
Classifier metamorphic relations"
MR | Content | Specific operation |
---|---|---|
MR-1 | Addition of uninformative attributes. | Add a column of irrelevant items. The data in this column is an unrelated string. This column should not affect the result. |
MR-2 | Permutation of the attribute. | Exchange any two columns of data, the training results should be unchanged. |
MR-3 | Additional training sample. | Copy a column, the training results should be unchanged. |
MR-4 | Permutation of class labels. | Replace the class label in the result column, and the training result should correspond to the transformation. |
MR-5 | Removal of classes. | Suppose that the classification result of the test sample ts is li, delete all samples of a certain class of the training set that is not li, and the classification result of ts is still li, and remains unchanged. |
Table 4
The consistency verification results of the one-dimensional metamorphic relations"
MR | Number | μ1 | μ2 | μ3 | μ4 | μ5 | μ6 | μ7 | μ8 | μ9 | μ10 | μ11 | μ12 | μ13 | μ14 | KR |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
MR-1 | 30 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0% |
MR-2 | 30 | 25 | 23 | 0 | 0 | 25 | 25 | 6 | 20 | 13 | 0 | 6 | 6 | 25 | 25 | 79% |
MR-3 | 24 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0% |
MR-4 | 30 | 25 | 25 | 0 | 0 | 25 | 25 | 0 | 10 | 5 | 10 | 0 | 0 | 25 | 25 | 71% |
MR-5 | 6 | 3 | 3 | 0 | 0 | 3 | 3 | 0 | 0 | 4 | 1 | 0 | 0 | 3 | 3 | 57% |
Total | - | 3 | 3 | 0 | 0 | 3 | 3 | 1 | 2 | 3 | 2 | 1 | 1 | 3 | 3 | 86% |
Table 5
The consistency verification results of the compositional metamorphic relations"
MR | Number | μ1 | μ2 | μ3 | μ4 | μ5 | μ6 | μ7 | μ8 | μ9 | μ10 | μ11 | μ12 | μ13 | μ14 | KR |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
MR-1.1 | 150 | 0 | 0 | 28 | 30 | 0 | 0 | 30 | 28 | 12 | 26 | 30 | 30 | 0 | 0 | 57% |
MR-1.2 | 150 | 116 | 98 | 0 | 0 | 116 | 116 | 24 | 94 | 84 | 0 | 24 | 24 | 116 | 116 | 79% |
MR-1.3 | 150 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0% |
MR-1.4 | 150 | 125 | 125 | 0 | 0 | 125 | 125 | 0 | 50 | 25 | 50 | 0 | 0 | 125 | 125 | 64% |
MR-1.5 | 30 | 10 | 11 | 20 | 20 | 10 | 10 | 20 | 16 | 17 | 17 | 20 | 20 | 10 | 10 | 100% |
MR-2.2 | 150 | 95 | 88 | 0 | 0 | 95 | 95 | 20 | 76 | 92 | 0 | 20 | 20 | 95 | 95 | 79% |
MR-2.3 | 120 | 100 | 92 | 0 | 0 | 100 | 100 | 24 | 80 | 52 | 0 | 24 | 24 | 100 | 100 | 79% |
MR-2.4 | 150 | 130 | 135 | 0 | 0 | 130 | 130 | 30 | 100 | 70 | 50 | 30 | 30 | 130 | 130 | 86% |
MR-2.5 | 30 | 5 | 5 | 25 | 25 | 5 | 5 | 25 | 20 | 20 | 20 | 25 | 25 | 5 | 5 | 100% |
MR-3.3 | 120 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0% |
MR-3.4 | 120 | 100 | 100 | 0 | 0 | 100 | 100 | 0 | 40 | 20 | 40 | 0 | 0 | 100 | 100 | 64% |
MR-3.5 | 24 | 9 | 10 | 15 | 15 | 9 | 9 | 15 | 12 | 13 | 13 | 15 | 15 | 9 | 9 | 100% |
MR-4.4 | 150 | 116 | 116 | 117 | 117 | 116 | 116 | 117 | 120 | 116 | 117 | 117 | 117 | 116 | 116 | 100% |
MR-4.5 | 6 | 4 | 4 | 5 | 5 | 4 | 4 | 5 | 4 | 5 | 5 | 5 | 5 | 4 | 4 | 100% |
MR-5.5 | 18 | 8 | 8 | 2 | 3 | 8 | 8 | 3 | 1 | 7 | 11 | 11 | 3 | 8 | 8 | 100% |
Total | - | 12 | 12 | 7 | 7 | 12 | 12 | 11 | 13 | 13 | 10 | 11 | 11 | 12 | 12 | 100% |
1. | Z. Markel and M. Bilzor, “Building a Machine Learning Classifier for Malware Detection,” inProceedings of 2014 Second Workshop on Anti-malware Testing Research (WATeR), pp. 1-4, Canterbury, 2014 |
2. | M. Lei, L.Yang, Z. J. Jun, W. Y. Dong, J. X. Felix, Z. F. Yuan, et al.,“DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems,” inProceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering (ASE), pp. 120-131, Montpellier, France, September 2018 |
3. | X. X. Yuan, W. K. H. Joshua, M. Christian, K. Gail, X. B. Wen, and C. Y. Tsong, “Testing and Validating Machine Learning Classifiers by Metamorphic Testing,” Journal of Systems and Software (online since December 7, 2010))(DOI 10.1016/S0164121210003213) |
4. | N. Shin and B. N. Hai, “Dataset Coverage for Testing Machine Learning Computer Programs,” inProceedings of 2016 23rd Asia-Pacific Software Engineering Conference (APSEC), pp. 297-304, Hamilton, 2016 |
5. | S. Al-Azani and J. Hassine,“Validation of Machine Learning Classifiers using Metamorphic Testing and Feature Selection Techniques,” inProceedings of International Workshop on Multi-disciplinary Trends in Artificial Intelligence (MIWAI), pp. 77-91, Springer, Cham, 2017 |
6. | A. Ozcift and A. Gulten,“Classifier Ensemble Construction with Rotation Forest to Improve Medical Diagnosis Performance of Machine Learning Algorithms,” Computer Methods and Programs in Biomedicine(online since March 2011)2011.03.018) |
7. | L. R. Chao, L. X. Ming, H. Song,H. Z. Wei,“Metamorphic Relation Construction Method based on Compositional Function,” Command Information System and Technology(online since February 2013)2013.01.016) |
8. | D. G. Wei, X. B. Wen, C. Lin, N. C. Hai,W. L. Lu,“Cases Studies on Testing with Compositional Metamorphic Relations,” Journal of Southeast University(online since December 2008)2008.04.009) |
9. | L. Huai, L. Xuan,C. T. Yueh, “A New Method for Constructing Metamorphic Relations,” inProceedings of 2012 12th International Conference on Quality Software (QSIC), pp. 59-68, Xi'an, China, October 2012 |
10. | P. Harrington.Dection Tree, In Machine Learning in Action, People Post Press, pp. 32-52, 2013 |
11. | Z. Jie, W. Z. Yi, Z. L. Ming,H. Dan,“Predictive Mutation Testing,” IEEE Transactions on Software Engineering (online since July2016) (DOI 10.1145/2931037.2931038) |
12. | Z. Jing, H. X. Gang,Z. Bin,“Test Approach for the Program of Clusters based on Metamorphic Relations,” Journal of Electronic Measurement and Instrument(online since August 2011)2011.00688) |
13. | Benoit Julien, “UCI machine learning repository,” |
14. | M. Bellare, A. Boldyreva,A. Palacio, “An Uninstantiable Random-Oracle-Model Scheme for a Hybrid-Encryption Problem,” inProceedings of International Conference on the Theory and Applications of Cryptographic Techniques (EUROCRYPT), pp. 171-188, Heidelberg, Germany, 2014 |
15. | J. Vondrák, “Optimal Approximation for the Submodular Welfare Problem in the Value Oracle Model,” inProceedings of the 40th Annual ACM Symposium on Theory of Computing (STOC), pp. 67-74, British, Canada, May 2008 |
16. | A. Ozcift and A. Gulten, “Classifier Ensemble Construction with Rotation Forest to Improve Medical Diagnosis Performance of Machine Learning Algorithms,” Computer Methods and Programs in Biomedicine(online since March 2011)2011.03.018) |
17. | A. Gotlieb and B. Botella, “Automated Metamorphic Testing,” inProceedings of 26th Annual International Computer Software and Applications Conference (COMPSAC), pp. 34-40, December 2003 |
18. | C. T. Ysong, F. J. Qiang,T. H. Tse, “Metamorphic Testing of Programs on Partial Differential Equations: a Case Study,” inProceedings of 26th Annual International Computer Software and Applications (COMPSAC), pp. 327-333, February 2002 |
19. | W. Wei and Z. K. Rong, “Researches on Basic Citerion and Strategy of Constructing Metamorphic Relations,” Computer Science, Vol. 39, No. 1, pp. 115-119, January 2012 |
[1] | Shikha Singh, Sumit Badotra, and Nitin Arvind Shelke. Applying Machine Learning Techniques for Comparative Analysis of Various Diseases [J]. Int J Performability Eng, 2024, 20(6): 379-390. |
[2] | Naveen Monga and Parveen Sehgal. Effective Software Defect Prediction: Evaluating Classifiers and Feature Selection with Firefly Algorithm [J]. Int J Performability Eng, 2024, 20(4): 195-204. |
[3] | K. Eswara Rao, Bala Murali Pydi, T. Panduranga Vital, P. Annan Naidu, U. D. Prasann, and T. Ravikumar. An Advanced Machine Learning Approach for Student Placement Prediction and Analysis [J]. Int J Performability Eng, 2023, 19(8): 536-546. |
[4] | Zhaoyue Zhang, An Zhang, Cong Sun, and Shanmei Li. Forecasting Airport Surface Traffic Congestion based on Decision Tree [J]. Int J Performability Eng, 2020, 16(5): 738-746. |
[5] | Sizhe Huang, Huosheng Xu, and Xuezhi Xia. Imbalanced Remote Sensing Ship Image Classification [J]. Int J Performability Eng, 2019, 15(6): 1709-1715. |
[6] | Hongbin Wang, Lei Hu, Xiaodong Xie, Lianke Zhou, and Huafeng Li. Incremental Integration Algorithm based on Incremental RLID3 [J]. Int J Performability Eng, 2019, 15(1): 252-260. |
[7] | Xie Wei, Xu Huoxi, Peng Liping, and Lan Zhigao. Short-Range Wireless Network Communication and Application based on Decision Tree Algorithm [J]. Int J Performability Eng, 2018, 14(11): 2561-2573. |
|