Removing spatially variant motion blur from a blurry image is a challenging problem as image blur can be complicated and difficult to model accurately. Recent progress in deep neural networks suggests that kernel-free single image deblurring can be achieved, but questions about deblurring performance persist. To improve performance, we proposed a deep convolutional neural network to restore a sharp image from a noisy/blurry image pair captured in quick succession. Two neural network structures, Deblur Long Short-Term Memory (LSTM) and DeblurMerger, are presented to fuse the pair of images in either sequential or parallel manner. To boost the training, gradient loss, adversarial loss, and spectral normalization are leveraged. The training dataset that consists of pairs of noisy/blurry images and the corresponding ground truth sharp image is synthesized based on the benchmark dataset GOPRO. We evaluated the trained networks on a variety of synthetic datasets and real image pairs. The results demonstrate that the proposed approach outperforms the state-of-the-art methods both qualitatively and quantitatively. DeblurLSTM achieves the best debluring performance, while DeblurMerger achieves nearly the same result but with significantly less computation time.