In this paper, hybrid chromosome genetic algorithm is applied to establishing the real-time classification model for surface defects in a large-scale strip steel image collection. After image preprocessing, four types of visual features, comprising geometric feature, shape feature, texture feature and grayscale feature, are extracted from the defect target image and its corresponding preprocessed image. In order to use genetic algorithm to optimize classification model based on hybrid chromosome, the structure of hybrid chromosome is designed to seamlessly integrate the kernel function, visual features and model parameters. And then the chromosome and the SVM classification model will be evolved and optimized according to the genetic operations and the fitness evaluation. In the end, the final SVM classifier is established using the decoding result of the optimal chromosome. The experimental results show that our method is effective and efficient in classifying the surface defects in a large-scale strip steel image collection.