Selective encryption algorithms have emerged as a popular technique for protecting the privacy of images during real-time transmission. For selectively encrypted images, it is necessary to evaluate their security and usability with visual security indices, and there have been a series of studies in this area. However, those proposed visual security indices (VSI) are often ineffective. We present a multi-directional structure and content-aware features-based visual security index (MCVSI) to perform an objective assessment on selectively encrypted images. Considering that selectively encrypted images prevent the main contents from being easily identified, stable local features in the images are extracted to indicate the degree of image content leakage. Meanwhile, we extract spatial structure information that closely aligns with human visual perception to indicate the level of variation in the overall image skeleton. Next, these features are subjected to similarity measurements to produce two types of similarity, content perception feature similarity and structure feature similarity. Finally, our visual security index is built by connecting all feature similarities and their corresponding visual security scores using the regression module. The experimental results and analyses indicate that the proposed MCVSI outperforms many existing mainstream VSI in terms of higher performance and stronger robustness, particularly on low and medium quality images.