Deep learning based defect detection methods require a large amount of high-quality defect data. However, the defect samples obtained in practical production are relatively expensive and lack diversity. The data generation method based on generative adversarial networks (GANs) can address the issue of insufficient defect samples at a lower cost. However, the training process of data generation algorithms based on GANs may be affected by various factors, making it challenging to ensure the stability of the quality of the synthesized defect data. Since high-quality defect data determine the performance and representation of the detection model, it is necessary to conduct alternative evaluations on the synthesized defect data. We comprehensively consider the evaluation indicators that affect the generated defect data and propose an alternative evaluation method for comprehensive data on surface defects of industrial products. First, an evaluation index system is constructed based on the attributes of defect data. Then, a substitution evaluation model for surface defect data is built using a multi-level quantitative analytic hierarchy process. Finally, to verify the effectiveness of the evaluation model, we use three advanced defect detection networks and validate the effectiveness of the evaluation model through comparative experiments. We provide an effective solution for screening high-quality defect data generation and improve the performance of downstream task defect detection models.