Convolutional neural networks (CNNs) have shown tremendous success for hyperspectral image classification in recent years. CNNs are capable of capturing multi-scale spectral–spatial characteristics of hyperspectral image pixels leading to good classification results. Despite the good accuracy, most of the classifiers misclassify some pixels and generate noisy classification maps. A deep CNN and Markov random field (MRF)-based two-stage classification framework is developed for hyperspectral images. The input image is first classified with the help of a deep CNN classifier. The results provided by CNN are further refined by applying stochastic relaxation labeling using MRF on the first-stage classification map to produce a refined classification map with better accuracy. This two-stage classification approach is particularly helpful if smaller misclassified regions are generated during the first-stage classification. Experiments are performed on one satellite-borne and three airborne hyperspectral images: Dioni, Indian Pines, Pavia University, and Salinas. The results show that the proposed method yields good classification accuracy and smoothed classification maps. The refinement by MRF relaxation improved the overall classification accuracy of the first-stage classifier by more than 2% for all the images. The overall classification accuracy in terms of κ coefficient is obtained as 0.9844, 0.9678, 0.9843, and 0.9841 for Dioni, Indian Pines, Pavia University, and Salinas images, respectively, which is comparable or better than several existing methods.