KEYWORDS: Machine learning, Image quality, Monte Carlo methods, Signal to noise ratio, Target detection, Image analysis, Image sensors, Image quality standards, Detection and tracking algorithms
An objective, quantitative method for assessing image interpretability for machine learning (ML) would be a valuable tool to support sensor design, collection management, and algorithm selection. The National Imagery Interpretability Rating Scale (NIIRS) has served as a useful standard for image analysis in support of intelligence, surveillance, and reconnaissance (ISR) missions. However, NIIRS focuses on human perception and empirical studies have demonstrated a tenuous relationship, at best, between NIIRS and observed performance for ML algorithms. We propose a new approach that approximates the Bayes error for object classification to establish an upper bound on ML performance for a given set of imagery. The process starts with high fidelity signatures from the object classes of interest. Degrading these signatures through an emulation of the sensor’s image chain produces signatures consistent with observed imagery from that sensor. Various distance metrics quantify the separability between specific object classes. We demonstrate a resampling technique to approximate the Bayes error, which is the theoretical limit for performance. This approach provides a quantitative measure that is independent of any specific machine learning model or methodology.
Image exploitation has evolved with a growing reliance on machine learning (ML). With increased reliance on ML comes questions about performance, reliability, and trust. When an imagery analyst performs analysis without the assistance of an ML model, the confidence is dependent on the quality of the underlying imagery and the expertise of the analyst. The National Imagery Interpretation Rating Scale (NIIRS) maps task complexity to image quality based on human cognition. Thus, NIIRS is an accepted standard for the information potential of the image for human analysis. Empirical analysis indicates that NIIRS is, at best, a partial indicator of expected performance for ML. This paper explores several factors that can affect the quality of ML-based results:
• The image quality as assessed in the ML context,
• The scene complexity of the image,
• The ML architecture, and
• The relationship between the training imagery and the mission imagery.
This paper will explore each of these factors and discuss their importance. We conclude with a set of recommendations for future research.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.