Accurate identification of cracks is of great significance for maintaining the health of the equipment. However, the low saliency of cracks in some composite or metal surfaces affects the detection accuracy of object detection algorithms. For example, small cracks on the inner surface of wind turbine blade (WTB) may be similar in color to the substrate or face complex background textures. Taking WTB cracks as low saliency crack samples, we propose a multimodal object detection convolutional neural network that fuses infrared images with visible images to detect cracks more accurately. The proposed network contains the CenterNet network with an existing fast and efficient mid-level fusion structure. First, we optimized the fusion structure to make it more suitable for extracting crack features. To address the problem that severe background interference in multimodal images affects the detection performance, we add channel attention to the fusion structure and train the improved network using a stepwise training method to enhance the framework’s ability to filter background interference information. Finally, the effectiveness of the improvements was verified by ablation experiments and feature map analysis, and the phenomena of wrong detection, missed detection, and repeated detection were reduced. The evaluation results show that the proposed multimodal object detection network is able to detect the low saliency WTB cracks more effectively, and the improvement of the network also results in a 6.22% increase in average precision. In addition, this method can be extended to other materials or scenes to identify very inconspicuous objects, replacing manual inspection in more challenging defect detection tasks.