Int J Performability Eng ›› 2020, Vol. 16 ›› Issue (1): 67-77.doi: 10.23940/ijpe.20.01.p8.6777

• Orginal Article • Previous Articles     Next Articles

Testing Machine Learning Classifiers based on Compositional Metamorphic Relations

Minghua Jia, Xiaodong Wang, Yue Xu, Zhanqi Cui*(), and Ruilin Xie   

  1. Computer School, Beijing Information Science and Technology University, Beijing, 100101, China
  • Submitted on ; Revised on ; Accepted on
  • Contact: Zhanqi Cui
  • Supported by:
    This work is supported by the National Natural Science Foundation of China (Grant No. 61702041), and the Science and Technology Project of Beijing Municipal Education Commission (Grant No. KM201811232016), the Qin Xin Talents Cultivation Program of Beijing Information Science & Technology University (Grant No. QXTCP C201906), and the Student Innovation Project of Beijing Information Science & Technology University (Grant No. 5101923400).


With the widespread application of intelligent software, more rigorous requirements are placed on the security and reliability of intelligent software programs. The major challenge is that the traditional test approaches cannot be easily adapted to the testing of intelligent software since the test oracle is not available and testing intelligent software needs to focus on training sets. The classifier is a typical intelligent software which has an uncertain output. As a result, the accuracy rate cannot be used to judge whether the classifier is defective. Therefore, this paper proposes an approach for testing classifiers based on compositional metamorphic relations. Initially, it was recommended to construct composite metamorphic relations to generate derivative test cases from the original test cases; followed by training classifier to predict the classification of the test data set; then checks their consistency between the original test cases and the derivative test cases against compositional metamorphic relations, and the detected violations are reported as bugs. The experiment is carried on the ID3 decision tree and compares the mutant detection capability with the on one-dimensional metamorphic testing. From the results received from the experiment conducted in this research shows that the proposed approach improves 16.7% of mutant kill rates.

Key words: intelligent software test, metamorphic test, mutation test, compositional metamorphic relations, classifiers, decision tree