Geospatial information is essential for development planning, like in the context of land and resource management. Existing research mainly focuses on multi-spectral or panchromatic images with specific sensor details. Incorporating multi-sensory panchromatic images at different scales makes the segmentation problem challenging. In this work, we propose a pixel-based globally trained model with a deep learning network to improve the segmentation results over existing patch-based networks. The proposed model consists of the encoder-decoder mechanism for semantic segmentation. Convolution and pooling layers are used at the encoding phase and transposed convolution and convolution layers are used for the decoding phase. Experiments show about 98.95% correct detection rate and 0.07% false detection rate of our proposed methodology on benchmark images. We prove the effectiveness of the proposed methodology by doing comparisons with previous work.