The recent convolutional neural network based studies on the compression artifact reduction (CAR) task have made great progress. However, most of these CAR methods still have some inadequacies. They are limited on the network capability due to treating extracted features equally and generate unpleasant visual results due to using the pixel-wise loss (e.g., L1/L2 loss) in training. Therefore, to address these issues, we propose a progressive multi-scale attention network (PMANet) for image CAR task and further introduce a PMANet-based generative adversarial network (PMAGAN) for visual quality improvement. Specifically, the key idea and the basic component of the PMANet is a multi-scale attention dense block, which effectively incorporates the multi-scale information to the model with the attention mechanism and thus enhances the network’s representation ability. The PMANet can be further improved with the designed progressive restoration structure. In addition, PMAGAN takes the PMANet as the generator and brings a generative adversarial networks framework with the adversarial training strategy. Experiments show that PMANet performs better than the state-of-the-art CAR methods, and PMAGAN can further achieve better visual quality with more natural and sharper textures.