Object detection has achieved good progress in the last few years. However, there are many challenges in the field of remote sensing imagery. Objects in remote sensing images usually have arbitrary orientations and various scales. In addition, some objects are easily overwhelmed by a cluttered background. To take advantage of single-stage object detectors that have fast speed, many cascaded structures based on single-stage detectors have been proposed to improve detection performance. However, feature inconsistency in cascade structure results in poor detection performance. To address these problems, we propose an innovative model in terms of both model improvement and loss function refinement. This model consists of an attention module to highlight useful information in cluttered scenes, a multi-scale feature fusion module, and a cascade refinement module with anchor constrained convolution to address feature inconsistency. Furthermore, Intersection-over-Union (IoU) classification loss is proposed to enhance the correlation between classification and localization, and a scale-aware regression loss is proposed to improve the detection performance on objects with different scales. We conducted extensive experiments on both the DOTA dataset and the HRSC2016 dataset, and the experimental results show that our model has advantages compared with current state-of-the-art methods.