In remote sensing images, the complex and changeable background often interferes with the detection of cloud and its shadow, which leads to the phenomenon of missing detection in areas with similar background colors. In addition, most methods cannot achieve the real-time detection effect because of too much calculation. To solve the above problems, the parallel asymmetric network with double attention is proposed. The algorithm adopts a parallel method, which allows two branches to participate in the calculation at the same time. One branch is used to fuse different information from two branches at different levels, and another branch is responsible for extracting deeper context information. At the same time, double attention module and asymmetric dilated block are used for two branches, respectively. Double attention module can help the algorithm pay more attention to the category and spatial information of cloud and its shadow, thus reducing the interference caused by background information in images. Asymmetric dilated block can extract two levels of receptive fields, and it can help the network to obtain enough receptive fields in cloud and its shadow images, thus reducing the cases of missed detection and false detection. Moreover, these two modules are lightweight in their parameters and calculation. Compared with some previous methods, our method can guarantee the accuracy and the detection speed is fast.