Millimeter-wave (MMW) imaging is a touch-free method for security inspection at railway stations and airports. However, automatic detection of dangerous objects in MMW images is challenging; the characteristics of images are remarkable with variable appearances of an object, different contrasts, and lower resolutions. We propose a fast and accurate concealed dangerous object detection method called MMW image Detection (MMWDet). MMWDet comprises two stages: single-image detection and multi-angle image detection. In the single-image detection stage, we design a more robust feature fusion relationship between adjacent feature maps in a feature pyramid network to achieve a more robust feature representation of an object. Meanwhile, we employ localization confidence rather than category confidence in the classification branch to address the negative sample interference in the training process. Then, multi-angle image detection is proposed to reduce the false and missed detection rate based on a single image by refining the detection results of multi-angle images, which maps and filters the prediction boxes frame by frame based on matching pairs of adjacent image feature points. We collected a MMW dataset that included 38k images. Extensive experiments on the MMW dataset demonstrate the effectiveness and improved performance of the proposed method compared with advanced general detection methods.