Convolutional neural network has been successfully applied to image denoising. In particular, dilated convolution, which expands the network’s receptive field, has been widely used and has achieved good results in image denoising. Losing some image information, a standard network cannot effectively reconstruct tiny image details from noisy images. To solve this problem, we propose a pyramid dilated CNN, which mainly has three pyramid dilated convolutional blocks (PDCBs) and a gated fusion unit (GFU). PDCB uses dilated convolution to expand the network’s receptive field and the pyramid structure to obtain more image details. GFU fuses and enhances the feature maps from different blocks. Experiments demonstrate that the proposed method outperforms the comparative state-of-the-art denoising methods for gray and color images. In addition, the proposed method can effectively deal with real-world noisy images.