Surface defects in hot-rolled steel strips are one of the common product problems for the steel industry, which harm the product appearance, affect the corrosion and wear resistance, and shorten the product service life. The natural defect samples are sparse, category imbalanced, and expensive manual annotations. Therefore, it is crucial to study the data augmentation and classification methods for small sample surface defects. To solve the above problems and improve the accuracy and real-time performance of defect classification, we propose a random offline data augmentation algorithm (Random-CutMix) and an improved MobileNet architecture (SP-MobileNet). The Random-CutMix algorithm expands the dataset by random sampling to balance the number of each defect class. The SP-MobileNet combines the inverse residual module with the channel shuffle mechanism (CSIn-Module) and pyramid split attention (PSA) module, which facilitates cross-group information flow and improves model representation capability and generalization performance with low computational cost. The accuracy, recall, F1 score, parameter, computational complexity, and frame rate of SP-MobileNet with Random-CutMix on the X-SDD dataset were 95.97%, 95.22%, 95.46%, 6.5 M, and 0.54 G, 72 FPS, respectively. The experiment results indicate that our method outperforms the state-of-the-art methods and provides an effective trade-off between accuracy and instantaneity in actual industrial production.