Selection of Test Sample to Improve DNN Test Efficiency based on Neuron Clusters

Abstract

Deep neural networks (DNNs) have complex structures and operation methods, making it difficult to pinpoint the cause of performance degradation. Moreover, DNNs are widely used in various domains, and the significance of their quality is emphasized. In the test process, selecting test samples that cause model misclassification can detect model vulnerabilities at an early stage, thereby improving test efficiency and reducing the time and cost required for testing. DNN testing faces the oracle problem, such as the manual labeling of large test datasets. Because manual labeling requires a significant amount of time and effort, it is important to select label-worthy test samples to minimize human effort and maximize advantages. This paper proposes a test sample selection method for improving test efficiency based on neuron clusters. Through experiments using public datasets and ResNet models, we verify whether neuron clusters can predict the model's classification results. We also compare the proposed method with coverage-based test sample selection methods in terms of the test efficiency. The neuron clusters predict the model's classification results with an accuracy of at least 73%. Furthermore, our proposed method is at least 7.5% more efficient than the coverage-based test sample selection methods.

Introduction

Research Questions

RQ1. Prediction

Prediction precision measurement

RQ2. Efficiency

Efficient test sample selection

Results

RQ1. Prediction

Comparison of model classification and neuron cluster prediction

RQ2. Efficiency

Number of selected test samples of neuron clusters and coverage-based methods on the MNIST dataset

Number of defects of neuron clusters and coverage-based methods on the MNIST dataset

Test efficiency of neuron clusters and coverage-based methods on the MNIST dataset

Number of selected test samples of neuron clusters and coverage-based methods on the Fashion-MNIST dataset

Number of defects of neuron clusters and coverage-based methods on the Fashion-MNIST dataset

Test efficiency of neuron clusters and coverage-based methods on the Fashion-MNIST dataset

Number of selected test samples of neuron clusters and coverage-based methods on the CIFAR-10 dataset

Number of defects of neuron clusters and coverage-based methods on the CIFAR-10 dataset

Test efficiency of neuron clusters and coverage-based methods on the CIFAR-10 dataset

Number of selected test samples of neuron clusters and coverage-based methods on the STL-10 dataset

Number of defects of neuron clusters and coverage-based methods on the STL-10 dataset

Test efficiency of neuron clusters and coverage-based methods on the STL-10 dataset