SignificanceHyperspectral microscopy grants the ability to characterize unique properties of tissues based on their spectral fingerprint. The ability to label and measure multiple molecular probes simultaneously provides pathologists and oncologists with a powerful tool to enhance accurate diagnostic and prognostic decisions. As the pathological workload grows, having an objective tool that provides companion diagnostics is of immense importance. Therefore, fast whole-slide spectral imaging systems are of immense importance for automated cancer prognostics that meet current and future needs.AimWe aim to develop a fast and accurate hyperspectral microscopy system that can be easily integrated with existing microscopes and provide flexibility for optimizing measurement time versus spectral resolution.ApproachThe method employs compressive sensing (CS) and a spectrally encoded illumination device integrated into the illumination path of a standard microscope. The spectral encoding is obtained using a compact liquid crystal cell that is operated in a fast mode. It provides time-efficient measurements of the spectral information, is modular and versatile, and can also be used for other applications that require rapid acquisition of hyperspectral images.ResultsWe demonstrated the acquisition of breast cancer biopsies hyperspectral data of the whole camera area within ∼1 s. This means that a typical 1 × 1 cm2 biopsy can be measured in ∼10 min. The hyperspectral images with 250 spectral bands are reconstructed from 47 spectrally encoded images in the spectral range of 450 to 700 nm.ConclusionsCS hyperspectral microscopy was successfully demonstrated on a common lab microscope for measuring biopsies stained with the most common stains, such as hematoxylin and eosin. The high spectral resolution demonstrated here in a rather short time indicates the ability to use it further for coping with the highly demanding needs of pathological diagnostics, both for cancer diagnostics and prognostics.
Hyperspectral (HS) images hold both spatial and spectral information of an imaged scene. This allows one to take advantage of the distinct spectral signatures of materials to perform classification tasks. Since HS data are also typically very large and redundant, it is appealing to utilize compressive sensing (CS) techniques for HS acquisition. CS avoids the need for postacquisition digital compression, as the compression is inherently performed electrooptically during acquisition. We research the performance of deep learning classification applied directly on the compressive measurements. We show that by using a spectral CS technique we previously developed, it is possible to reduce the captured data by an order of magnitude without significant loss in the classification performance.
The application of compressive sensing (CS) techniques for the hyperspectral (HS) imaging is very appealing since the acquisition of HS images is demanding in terms hardware and acquisition time, and since the application of CS framework matches well the HS imaging task, which involves capturing huge amount of typically very redundant data. During the last decade, we developed several CS HS imaging systems, which have demonstrated orders of magnitude reduction of the acquisition time and of storage requirements, improved signal-to-noise ratio, and reduction of the systems’ size and weight. In this paper we demonstrate how these systems can further benefit from employing deep learning (DL) tools for post-processing of the compressively sensed hyperspectral data. We overview some DL techniques that we have developed for improving the HS image reconstruction and target detection.
The utilization of compressive sensing (CS) techniques for hyperspectral (HS) imaging is appealing since HS data is typically huge and very redundant. The CS design offers a significant reduction of the acquisition effort, which can be manifested in faster acquisition of the HS datacubes, acquisition of larger HS images and removing the need for postacquisition digital compression. But, do all these benefits come at the expense of the ability to extract targets from the HS images? The answer to this question, of course, depends on the specific CS design and on the target detection algorithm employed. In a previous study we have shown that there is virtually no target detection performance degradation when a classical target detection algorithm is applied on data acquired with CS HS imaging techniques of the kind we have developed during the last years. In this paper we further investigate the robustness of our CS HS techniques for the task of object classification by deep learning methods. We show preliminary results demonstrating that deep neural network classifiers perform equally well when applied on HS data captured with our compressively sensed methods, as when applied on conventionally sensed HS data.
Hyperspectral imaging is applied in a wide range of defense, security and law enforcement applications. The spectral data caries valuable information for tasks such as identification, detection, and classification. However, the capturing of the spectral information, together with the spatial information, requires a significant acquisition effort. In the recent years we have developed several compressive hyperspectral imaging techniques demonstrating reduction of the captured data by at least an order of magnitude. However, compressive sensing techniques typically require computational heavy and time consuming iterative reconstruction algorithms. The computational burden is even more prominent in compressive spectral imaging due to the large amount of data involved. In this work we demonstrate the utilization of a convolutional neural network (CNN) for the reconstruction of spectral images captured with our Compressive Sensing -Miniature Ultraspectral Imager (CS-MUSI). We discuss the challenges of training the CNN for CS-MUSI and analyze the CNNbased reconstruction performance.
In the recent years, we have developed several architectures for compressive hyperspectral (HS) imagers. The compressive sensing (CS) design has allowed the reduction of the enormous acquisition effort associated with the huge dimensionality of the HS data. Unfortunately, the reduced sensing effort offered by the CS approach comes on the account of increased post-sensing computational burden. Conventional CS reconstruction involves algorithms that solve a ℓ1 minimization problem. Those algorithms are iterative and typically very computationally heavy. The computation burden is even more prominent when reconstructing 3D HS data, where each spectral image may have Gigavoxel size. Motivated by this, we have investigated replacing the CS iterative reconstruction step with an appropriate Deep Neural Network.
The parametrization of light rays in form of light fields (LF) have become the standard and probably the most common way for the representation, analysis and processing of rays emitted from 3D objects or from 3D displays. Essentially, the LFs are 4D maps representing the spatial and angular distribution of the intensity of the rays. Nowadays, with the increasing availability of spectral imagers, the conventional LF can be augmented with the spectral information, yielding to what we call spectral light fields (SLFs). Spectral light fields refer to a 5D distribution of spatial, angular and spectral ray’s distribution. Thus, the SLF can be viewed as spectral radiance over a 2D manifold, or as 5D parameterization of a plenoptic function. In this paper we show the utility of the SLFs for digital 3D reconstruction. We show that the additional spectral domain provides important information that can be utilized to overcome 3D reconstruction artefacts caused by ambiguities in commonly captured LFs. We demonstrate the utilization of the SLFs for profilomety and refocusing.
Light fields are four-dimensional parametrizations of rays, extensively used for the representation of the rays emitted from three-dimensional objects. With the increasing availability of spectral imagers, conventional light fields can be augmented by the additional spectral information, yielding a five-dimensional ray parametrization referred to as a spectral light field. We demonstrate that utilization of the spectral channel information can improve the profilometric performance of light-field cameras.
During the past years, several compressive spectral imaging techniques were developed. With these techniques, an optically compressed version of the spectral datacube is captured. Consequently, the information about the object and targets is captured in a lower dimensional space. A question that rises is whether the reduction of the captured space affects the target detection performance. The answer to this question depends on the compressive spectral imaging technique employed. In most compressive spectral imaging techniques, the target detection performance is deteriorated. We show that our recently introduced technique, dubbed Compressive Sensing Miniature Ultra-Spectral Imaging (CSMUSI), yields similar target detection and false detection rates to that of conventional hyperspectral cameras.
Recently we presented a Compressive Sensing Miniature Ultra-spectral Imaging System (CS-MUSI)1 . This system consists of a single Liquid Crystal (LC) phase retarder as a spectral modulator and a gray scale sensor array to capture a multiplexed signal of the imaged scene. By designing the LC spectral modulator in compliance with the Compressive Sensing (CS) guidelines and applying appropriate algorithms we demonstrated reconstruction of spectral (hyper/ ultra) datacubes from an order of magnitude fewer samples than taken by conventional sensors. The LC modulator is designed to have an effective width of a few tens of micrometers, therefore it is prone to imperfections and spatial nonuniformity. In this work, we present the study of this nonuniformity and present a mathematical algorithm that allows the inference of the spectral transmission over the entire cell area from only a few calibration measurements.
We review two compressive spectroscopy techniques based on modulation in the spectral domain that we have recently proposed. Both techniques achieve a compression ratio of approximately 10:1, however each with a different sensing mechanism. The first technique uses a liquid crystal cell as a tunable filter to modulate the spectral signal, and the second technique uses a Fabry-Perot etalon as a resonator. We overview the specific properties of each of the techniques.
Compressive sensing theory was proposed to deal with the high quantity of measurements demanded by traditional hyperspectral systems. Recently, a compressive spectral imaging technique dubbed compressive sensing miniature ultraspectral imaging (CS-MUSI) was presented. This system uses a voltage controlled liquid crystal device to create multiplexed hyperspectral cubes. We evaluate the utility of the data captured using the CS-MUSI system for the task of target detection. Specifically, we compare the performance of the matched filter target detection algorithm in traditional hyperspectral systems and in CS-MUSI multiplexed hyperspectral cubes. We found that the target detection algorithm performs similarly in both cases, despite the fact that the CS-MUSI data is up to an order of magnitude less than that in conventional hyperspectral cubes. Moreover, the target detection is approximately an order of magnitude faster in CS-MUSI data.
The interest in liquid crystal devices for photonic non-display devices has grown recently due to their mature quality and the continuous improvement of their speed combined with the rising nanoscale and optoelectronic technologies. Of particular interest is their application in imaging systems as compact devices to manipulate the wavefront, wavelength, phase or polarization.
Recently we have been developing variety of specially designed LC devices integrated into imaging systems for specific spectro-polarimetric imaging applications using small number of LC devices. These included: (i) wide dynamic range tunable filters for hyperspectral imaging and frequency domain optical coherence tomography, (ii) discrete narrowband tunable filter for multispectral imaging, (iii) compact polarization rotator for polarimetric imaging, (v) wideband achromatic waveplate for polarimetric camera, (vii) polarization independent LCFP tunable filter, and lately (vii) single LC retarder for hyperspectral imaging. In this report we shall present the main concepts of these devices and their functionality into spectro-polarimetric imaging systems such as in skin cancer diagnosis, and imaging oximetry [1-4].
Selected Publications:
1. S. Isaacs et.al, Applied Optics 53, H91-H101 (2014).
2. M. AbuLeil et.al., Optics Letter 39, 5487-90 (2014).
3. I. August, et.al., Scientific Reports, communicated 2016.
4. M. AbuLeil et.al., in preparation.
The theory of compressive sensing (CS) has opened up new opportunities in the field of imaging. However, its
implementation in this field is often not straight-forward and the optical imaging system engineer encounters
several hurdles on the way of compressive imaging (CI) realization. The principles of CI design may differ
drastically from the principles used for conventional imaging. Analytical tools developed for conventional imaging
may not be optimal for compressive imaging. Nor are the conventional imaging components. Therefore often the CI
designer needs to develop new tools, and imaging schemes. In this paper we overview the main challenges that
might arise in the design of compressive imaging systems. The challenges are demonstrated through four tasks and
systems: compressive two dimensional (2D) imager, compressive motion detection, compressive spectral imaging
and compressive holography.
Recently we introduced a Compressive Sensing Miniature Ultra-Spectral Imaging (CS-MUSI) system. The system is based on a single Liquid Crystal (LC) cell and a parallel sensor array where the liquid crystal cell performs spectral encoding. Within the framework of compressive sensing, the CS-MUSI system is able to reconstruct ultra-spectral cubes captured with only an amount of ~10% samples compared to a conventional system. Despite the compression, the technique is extremely complex computationally, because reconstruction of ultra-spectral images requires processing huge data cubes of Gigavoxel size. Fortunately, the computational effort can be alleviated by using separable operation. An additional way to reduce the reconstruction effort is to perform the reconstructions on patches. In this work, we consider processing on various patch shapes. We present an experimental comparison between various patch shapes chosen to process the ultra-spectral data captured with CS-MUSI system. The patches may be one dimensional (1D) for which the reconstruction is carried out spatially pixel-wise, or two dimensional (2D) - working on spatial rows/columns of the ultra-spectral cube, as well as three dimensional (3D).
Recently we introduced a hyperspectral compressive sensing scheme that uses separable projections in the spatial and spectral domains. The separable encoding schemes facilitates the optical implementation, reduces the computational burden dramatically, and storage requirements. Owing to these benefits we have been able to encode the hyperspectral cube in all three dimensions. In this work we present a comparison between various reconstructions methods applied to the hyperspectral data captured with our separable compressive sensing systems.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.