The wide application of agricultural greenhouses globally has brought economic benefits; however, it has also led to many environmental problems. The timely and accurate acquisition of greenhouse areas and distribution is valuable for authorities seeking to optimize regional agricultural management and mitigate environmental pollution. Automatic extraction of the greenhouses from high spatial resolution remote sensing (RS) imagery based on deep learning can reduce labor costs and improve operational efficiency to have better application prospects. In this paper, we propose a multi-channel fused fully convolutional network (FCN) optimized by the optimal scale object-oriented segmentation results for agricultural greenhouse extraction from high spatial resolution RS imagery. First, to make full use of remote sensing feature images of target objects and to not increase the complexity of the deep learning network, we constructed a decision-level fusion FCN network that can simultaneously input multiple remote sensing images for preliminary extraction of greenhouse. Second, to address a defect in the classical FCN network causing the easy loss of ground object details, we optimized the preliminary extraction results from FCN by the results of object-oriented segmentation. Finally, the optimized greenhouse extraction results were processed by the mathematical morphology, and the final extraction results were obtained. The experimental results demonstrate that: (1) Multi-channel fused FCN model could use the unique spectral characteristics of different ground objects. (2) Optimized initial extraction results from FCN based on the optimal scale object-oriented segmentation results could fully maintain the edge details of the greenhouse. Experimental results show that the proposed method can extract the greenhouse effectively. The precision and F value of our proposed method are 92.68% and 0.94.