We introduce a deep learning-based approach utilizing pyramid sampling for the automated classification of HER2 status in immunohistochemically (IHC) stained breast cancer tissue images. Our deep learning-based method leverages pyramid sampling to analyze features across multiple scales from IHC-stained breast tissue images, managing the computational load effectively and addressing the challenges of HER2 expression heterogeneity by capturing detailed cellular features and broader tissue architecture. Upon application to 523 core images, the model achieved a classification accuracy of 85.47%, demonstrating the ability to counteract staining variability and tissue heterogeneity, which might improve the accuracy and timeliness of breast cancer treatment planning.
We present a rapid, stain-free, and automated viral plaque assay utilizing deep learning and time-lapse holographic imaging, which can significantly reduce the time needed for plaque-forming unit (PFU) detection and entirely bypass the chemical staining and manual counting processes. Demonstrated with vesicular stomatitis virus (VSV), our system identified the first PFU events as early as 5 hours of incubation and detected >90% of PFUs with 100% specificity in <20 hours, saving >24 hours compared to the traditional viral plaque assays that take ≥48 hours. Furthermore, our method was proven to adapt seamlessly to new types of viruses by transfer learning.
We present a method for accurately performing complex-valued linear transformations with a Diffractive Deep Neural Network (D2NN) under spatially incoherent illumination. By employing 'mosaicing' and 'demosaicing' techniques, complex data are encoded into optical intensity patterns for all-optical diffractive processing, and then decoded back into the complex domain at the output aperture. This framework not only enhances the capabilities of D2NNs for visual computing tasks but also opens up new avenues for applications in image encryption under natural light conditions to demonstrate the potential of diffractive optical networks in modern visual information processing needs.
We present subwavelength imaging of amplitude- and phase-encoded objects based on a solid-immersion diffractive processor designed through deep learning. Subwavelength features from the objects are resolved by the collaboration between a jointly-optimized diffractive encoder and decoder pair. We experimentally demonstrated the subwavelength-imaging performance of solid immersion diffractive processors using terahertz radiation and achieved all-optical reconstruction of subwavelength phase features of objects (with linewidths of ~λ/3.4, where λ is the wavelength) by transforming them into magnified intensity images at the output field-of-view. Solid-immersion diffractive processors would provide cost-effective and compact solutions for applications in bioimaging, sensing, and material inspection, among others.
We demonstrate a reconfigurable diffractive deep neural network (termed R‑D2NN) with a single physical model performing a large set of unique permutation operations between an input and output field-of-view by rotating different layers within the diffractive network. Our study numerically demonstrated the efficacy of R‑D2NN by accurately approximating 256 distinct permutation matrices using 4 rotatable diffractive layers. We experimentally validated the proof-of-concept of reconfigurable diffractive networks using terahertz radiation and 3D-printed diffractive layers, achieving high concordance with numerical simulations. The reconfigurable design of R‑D2NN provides scalability with high computing speed and efficient use of materials within a single fabricated model.
We introduce an information hiding-decoding system, which employs a passive diffractive processor as the front-end and an electronic decoder as the back-end, offering a fast, energy-efficient, and scalable solution for protecting visual information. This diffractive processor all-optically transforms arbitrary input messages into deceptive output patterns, decipherable only through a jointly-trained electronic decoder neural network. This method can successfully hide infinitely many input messages into ordinary-looking patterns at its output field-of-view, which can be subsequently decoded by an electronic network. We experimentally validated the feasibility of our information-hiding camera by 3D-printing a physical diffractive system and testing it under terahertz illumination.
We present a fast virtual-staining framework for defocused autofluorescence images of unlabeled tissue, matching the performance of standard virtual-staining models using in-focus label-free images. For this, we introduced a virtual-autofocusing network to digitally refocus the defocused images. Subsequently, these refocused images were transformed into virtually-stained H&E images using a successive neural network. Using coarsely-focused autofluorescence images, with 4-fold fewer focus points and 2-fold lower focusing precision, we achieved equivalent virtual-staining performance to standard H&E virtual-staining networks that utilize finely-focused images, helping us decrease the total image acquisition time by ~32% and the autofocusing time by ~89% for each whole-slide image.
We present a diffractive network (D2NN) design to all-optically perform distinct transformations for different input data classes. This class-specific transformation D2NN processes the input optical field, generating the output optical field whose amplitude or intensity closely approximates the transformed/encrypted version of the input using a transformation matrix specific to the corresponding data class. The original information can be recovered only by applying the class-specific decryption keys to the corresponding class at the diffractive network's output field-of-view. The efficacy of the presented class-specific image encryption framework was validated both numerically and experimentally, tested at 1550 nm and 0.75 mm wavelengths.
We introduce a unidirectional imager that facilitates polarization-insensitive and broadband operation using isotropic, linear materials. This design comprises diffractive layers with hundreds of thousands of learnable phase features, trained using deep learning to enable power-efficient, high-fidelity imaging in the forward direction (A-to-B), while simultaneously inhibiting optical transmission and image formation in the reverse direction (B-to-A). We experimentally tested our designs using terahertz radiation, providing a good match with our simulations. Furthermore, we demonstrated a wavelength-selective unidirectional imager that performs unidirectional imaging along A-to-B at a predetermined wavelength, while at a second wavelength, the unidirectional operation switches from B-to-A.
Diffractive deep neural networks utilize successive, spatially-engineered diffractive surfaces trained via deep learning to all-optically process input optical fields based on a desired transformation. We present the design of a broadband diffractive network that can all-optically perform a large set of arbitrary complex-valued linear transformations, wherein the input/output data are encoded at W different wavelength channels, each assigned to a unique linear transformation, covering, e.g., W>100-2000. This broadband diffractive visual processor may foster the development of all-optical visual processors with substantial data bandwidth and parallel computation capabilities, creating intelligent machine vision systems for all-optical processing of multi-color or hyperspectral objects/scenes.
We report deep learning-based design of diffractive all-optical processors for performing arbitrary linear transformations of optical intensity under spatially incoherent illumination. We show that a diffractive optical processor can approximate an arbitrary linear intensity transformation under spatially incoherent illumination with a negligible error if it has a sufficient number of optimizable phase-only diffractive features distributed over its diffractive surfaces. Our analysis and design framework could open up new avenues in designing incoherent imaging systems with an arbitrary set of spatially-varying point-spread functions (PSFs). Moreover, this framework can also be extended to design task-specific all-optical visual information processors under natural illumination.
We present an approach for quantitative phase imaging (QPI) through random, unknown phase diffusers using a diffractive optical network consisting of successive layers optimized through deep learning. Unlike traditional digital reconstruction methods, our all-optical diffractive processor requires no external power beyond the illumination light and completes its QPI reconstruction as the light is transmitted through a thin diffractive processor. With its low power consumption, high frame rate, and compact size, our design offers a transformative alternative for QPI through random, unknown phase diffusers, and it can be readily scaled to work at different wavelengths for various applications in biomedical imaging.
KEYWORDS: Free space optics, Diffusers, Education and training, Deep learning, 3D modeling, Optical transmission, Neural networks, Mathematical optimization, Light sources and illumination, Image transmission
We report an optical diffractive decoder with an electronic encoder network to facilitate the accurate transmission of optical information of interest through unknown random phase diffusers along the optical path. This hybrid electronic-optical model was trained via supervised learning, and comprises a convolutional neural network-based encoder and jointly-trained passive diffractive layers. After their joint-training using deep learning, our hybrid model can accurately transfer optical information even in the presence of unknown phase diffusers, generalizing to new random diffusers never seen before. We experimentally validated this framework using a 3D-printed diffractive network, axially spanning <70λ, where λ=0.75mm is the illumination wavelength.
As an optical processor, a diffractive deep neural network (D2NN) utilizes engineered diffractive surfaces designed through machine learning to perform all-optical information processing, completing its tasks at the speed of light propagation through thin optical layers. With sufficient degrees of freedom, D2NNs can perform arbitrary complex-valued linear transformations using spatially coherent light. Similarly, D2NNs can also perform arbitrary linear intensity transformations with spatially incoherent illumination; however, under spatially incoherent light, these transformations are nonnegative, acting on diffraction-limited optical intensity patterns at the input field of view. Here, we expand the use of spatially incoherent D2NNs to complex-valued information processing for executing arbitrary complex-valued linear transformations using spatially incoherent light. Through simulations, we show that as the number of optimized diffractive features increases beyond a threshold dictated by the multiplication of the input and output space-bandwidth products, a spatially incoherent diffractive visual processor can approximate any complex-valued linear transformation and be used for all-optical image encryption using incoherent illumination. The findings are important for the all-optical processing of information under natural light using various forms of diffractive surface-based optical processors.
We present a novel approach to perform quantitative phase imaging (QPI) through random phase diffusers using a diffractive neural network consisting of successive diffractive layers optimized using deep learning. This diffractive network is trained to convert the phase information of samples positioned behind random diffusers into intensity variations at the output, enabling all-optical phase recovery and quantitative phase imaging of objects hidden by unknown random diffusers. Unlike traditional digital image reconstruction methods, our all-optical diffractive processor does not require external power beyond the illumination beam and operates at the speed of light propagation.
We explore the parallel information processing capacity of a broadband diffractive optical network and demonstrate that a single diffractive network could perform a large group of arbitrarily-selected, complex-valued linear transformations between its input and output fields-of-view at different wavelengths, accessed sequentially or simultaneously. Through deep learning-based training of the thickness values of its diffractive features, we demonstrate that a wavelength-multiplexed diffractive processor can implement W>180 complex-valued linear transformations with a negligible error when its number of trainable diffractive features approaches 2W×I×O, where I and O refer to the number of input and output pixels, respectively.
We present data class-specific transformation diffractive networks that all-optically perform different preassigned transformations for different input data classes. The visual information encoded in the amplitude, phase, or intensity channel of the input field is all-optically processed and transformed/encrypted by the diffractive network. The amplitude or intensity of the resulting field approximates the transformed/encrypted input information using the transformation matrix specifically assigned for that data class. We experimentally validated this class-specific transformation framework by designing and fabricating two diffractive networks at 1550nm and 0.75mm wavelengths. The presented framework provides a fast, secure, and energy-efficient solution to data encryption applications.
We present the first demonstration of unidirectional imaging that permits image formation along only one direction, from an input field-of-view to an output field-of-view, while eliminating optical transmission in the reverse direction. This unidirectional imager is formed by diffractive layers composed of isotropic linear materials spatially-coded with thousands of phase features optimized using deep learning. We experimentally tested our diffractive design using a terahertz setup and 3D-printed diffractive layers, which revealed a good agreement with our numerical simulations. The designs of these diffractive unidirectional imagers are compact and can be scaled to operate at different parts of the electromagnetic spectrum.
Free-space optical information transfer through diffusive media is critical in many applications, such as biomedical devices and optical communication, but remains challenging due to random, unknown perturbations in the optical path. We demonstrate an optical diffractive decoder with electronic encoding to accurately transfer the optical information of interest, corresponding to, e.g., any arbitrary input object or message, through unknown random phase diffusers along the optical path. This hybrid electronic-optical model, trained using supervised learning, comprises a convolutional neural network-based electronic encoder and successive passive diffractive layers that are jointly optimized. After their joint training using deep learning, our hybrid model can transfer optical information through unknown phase diffusers, demonstrating generalization to new random diffusers never seen before. The resulting electronic-encoder and optical-decoder model was experimentally validated using a 3D-printed diffractive network that axially spans <70λ, where λ = 0.75 mm is the illumination wavelength in the terahertz spectrum, carrying the desired optical information through random unknown diffusers. The presented framework can be physically scaled to operate at different parts of the electromagnetic spectrum, without retraining its components, and would offer low-power and compact solutions for optical information transfer in free space through unknown random diffusive media.
We present a diffractive camera that performs class-specific imaging of target objects, while all-optically and instantaneously erasing the objects from other classes during light propagation through thin diffractive layers, maximizing privacy preservation. We experimentally validated this class-specific camera design by 3D-printing the resulting diffractive layers (optimized through deep learning) and selectively imaging MNIST handwritten digits using the assembled camera system under terahertz radiation. The presented object class-specific camera is passive and does not require external computing power, providing a data-efficient solution to task-specific and privacy-aware modern imaging applications.
We report a computer-free imaging framework in which a set of transmissive diffractive layers were trained using deep learning to all-optically reconstruct arbitrary objects hidden by unknown, random phase diffusers. The image reconstruction of the object hidden behind a random and unknown phase diffuser is completed at the speed of light propagation through a thin, engineered diffractive volume. Our analyses provide a comprehensive guide for designing robust and generalizable diffractive imagers to all-optically see through random diffusers, which might be transformative for various fields, such as biomedical imaging, atmospheric physics, and autonomous driving.
We report an all-optical object classification framework using a single-pixel diffractive network and spectrum encoding, classifying unknown objects through unknown random phase diffusers at the speed of light. Using this single-pixel diffractive network design, we numerically achieved a blind testing accuracy of 88.53%, classifying unknown handwritten digits through 80 unknown random diffusers that were never used during training. This framework presents a time- and energy-efficient all-optical solution for directly sensing through unknown random diffusers using a single pixel and will be of broad interest to various fields, such as security, biosensing and autonomous driving.
We report label-free, in vivo virtual histology of skin using reflectance confocal microscopy (RCM). We trained a deep neural network to transform in vivo RCM images of unstained skin into virtually stained H&E-like microscopic images with nuclear contrast. This framework successfully generalized to diverse skin conditions, e.g., normal skin, basal cell carcinoma, and melanocytic nevi, as well as distinct skin layers, including the epidermis, dermal-epidermal junction, and superficial dermis layers. This label-free in vivo skin virtual histology framework can be transformative for faster and more accurate diagnosis of malignant skin neoplasms, with the potential to significantly reduce unnecessary skin biopsies.
We present a virtual staining framework that can rapidly stain defocused autofluorescence images of label-free tissue, matching the performance of standard virtual staining models that use in-focus unlabeled images. We trained and blindly tested this deep learning-based framework using human lung tissue. Using coarsely-focused autofluorescence images acquired with 4× fewer focus points and 2× lower focusing precision, we achieved equivalent performance to the standard virtual staining that used finely-focused autofluorescence input images. We achieved a ~32% decrease in the total image acquisition time needed for virtual staining of a label-free whole-slide image, alongside a ~89% decrease in the autofocusing time.
We present a deep learning-based framework to virtually transfer images of H&E-stained tissue to other stain types using cascaded deep neural networks. This method, termed C-DNN, was trained in a cascaded manner: label-free autofluorescence images were fed to the first generator as input and transformed into H&E stained images. These virtually stained H&E images were then transformed into Periodic acid–Schiff (PAS) stain by the second generator. We trained and tested C-DNN on kidney needle-core biopsy tissue, and its output images showed better color accuracy and higher contrast on various histological features compared to other stain transfer models.
We present a high-throughput and automated system for the early detection and classification of bacterial colony-forming units (CFUs) using a thin-film transistor (TFT) image sensor. A lens-free imager was built using the TFT sensor with a ~7 cm2 field-of-view to collect the time-lapse images of bacterial colonies. Two trained neural networks were used to detect and classify the bacterial colonies based on their spatio-temporal features. Our system achieved an average CFU detection rate of 97.3% at 9 hours of incubation and an average CFU recovery rate of 91.6% at ~12 hours, saving ~12 hours compared to the EPA-approved method.
We present a stain-free, rapid, and automated viral plaque assay using deep learning and holography, which needs significantly less sample incubation time than traditional plaque assays. A portable and cost-effective lens-free imaging prototype was built to record the spatio-temporal features of the plaque-forming units (PFUs) during their growth, without the need for staining. Our system detected the first cell lysing events as early as 5 hours of incubation and achieved >90% PFU detection rate with 100% specificity in <20 hours, saving >24 hours compared to the traditional viral plaque assays that take ≥48 hours.
We present a virtual immunohistochemical (IHC) staining method based on label-free autofluorescence imaging and deep learning. Using a trained neural network, we transform multi-band autofluorescence images of unstained tissue sections to their bright-field equivalent HER2 images, matching the microscopic images captured after the standard IHC staining of the same tissue sections. Three pathologists’ blind evaluations of HER2 scores based on virtually stained and IHC-stained whole slide images revealed the statistically equivalent diagnostic values of the two methods. This virtual HER2 staining method provides a rapid, accurate, and low-cost alternative to the standard IHC staining methods and allows tissue preservation.
Large-scale linear operations are the cornerstone for performing complex computational tasks. Using optical computing to perform linear transformations offers potential advantages in terms of speed, parallelism, and scalability. Previously, the design of successive spatially engineered diffractive surfaces forming an optical network was demonstrated to perform statistical inference and compute an arbitrary complex-valued linear transformation using narrowband illumination. We report deep-learning-based design of a massively parallel broadband diffractive neural network for all-optically performing a large group of arbitrarily selected, complex-valued linear transformations between an input and output field of view, each with Ni and No pixels, respectively. This broadband diffractive processor is composed of Nw wavelength channels, each of which is uniquely assigned to a distinct target transformation; a large set of arbitrarily selected linear transformations can be individually performed through the same diffractive network at different illumination wavelengths, either simultaneously or sequentially (wavelength scanning). We demonstrate that such a broadband diffractive network, regardless of its material dispersion, can successfully approximate Nw unique complex-valued linear transforms with a negligible error when the number of diffractive neurons (N) in its design is ≥2NwNiNo. We further report that the spectral multiplexing capability can be increased by increasing N; our numerical analyses confirm these conclusions for Nw > 180 and indicate that it can further increase to Nw ∼ 2000, depending on the upper bound of the approximation error. Massively parallel, wavelength-multiplexed diffractive networks will be useful for designing high-throughput intelligent machine-vision systems and hyperspectral processors that can perform statistical inference and analyze objects/scenes with unique spectral properties.
Immunohistochemical (IHC) staining of the human epidermal growth factor receptor 2 (HER2) is routinely performed on breast cancer cases to guide immunotherapies and help predict the prognosis of breast tumors. We present a label-free virtual HER2 staining method enabled by deep learning as an alternative digital staining method. Our blinded, quantitative analysis based on three board-certified breast pathologists revealed that evaluating HER2 scores based on virtually-stained HER2 whole slide images (WSIs) is as accurate as standard IHC-stained WSIs. This virtual HER2 staining can be extended to other IHC biomarkers to significantly improve disease diagnostics and prognostics.
Reflectance confocal microscopy (RCM) can provide in vivo images of the skin with cellular-level resolution; however, RCM images are grayscale, lack nuclear features and have a low correlation with histology. We present a deep learning-based virtual staining method to perform non-invasive virtual histology of the skin based on in vivo, label-free RCM images. This virtual histology framework revealed successful inference for various skin conditions, such as basal cell carcinoma, also covering distinct skin layers, including epidermis and dermal-epidermal junction. This method can pave the way for faster and more accurate diagnosis of malignant skin neoplasms while reducing unnecessary biopsies.
We present a deep learning-aided imaging system for early detection and classification of live bacterial colonies by capturing time-lapse holographic images of an agar plate and analyzing these images using deep neural networks. We blindly tested our system by identifying Escherichia coli and total coliform bacteria in spiked water samples and successfully detected 90% of the bacterial colonies within 7-10 h, while keeping 99.2~100% precision. We further classified the corresponding species within 7.6-12 h of incubation with 80% accuracy, which represents >12 h time-savings. Our system also achieved a limit-of-detection of ~1 CFU/L within 9 h of total test time.
We present a deep learning-based high-throughput cytometer to detect rare cells in whole blood using a cost-effective and light-weight design. This system uses magnetic-particles to label and enrich the target cells. Then, a periodically-alternating magnetic-field creates time-modulated diffraction patterns of the target cells that are recorded using a lensless microscope. Finally, a custom-designed convolutional network is used to detect and classify the target cells based on their modulated spatio-temporal patterns. This cytometer was tested with cancer cells spiked in whole blood to achieve a limit-of-detection of 10 cells/mL. This compact, cost-effective and high-throughput cytometer might serve diagnostics needs in resource-limited-settings.
We present a deep learning-enabled holographic polarization microscope that only requires one polarization state to image/quantify birefringent specimen. This framework reconstructs quantitative birefringence retardance and orientation images from the amplitude/phase information obtained using a lensless holographic microscope with a pair of polarizer and analyzer. We tested this technique with various birefringent samples including monosodium urate and triamcinolone acetonide crystals to demonstrate that the deep network can accurately reconstruct the retardance and orientation image channels. This method has a simple optical design and presents a large field-of-view (>20-30mm2), which might broaden the access to advanced polarization microscopy techniques in low-resource-settings.
We present a high-throughput and cost-effective computational cytometer for rare cell detection, where the target cells are specifically labeled with magnetic particles and exhibit an oscillatory motion under a periodically-changing magnetic field. The time-varying diffraction patterns of the oscillating cells are then captured with a holographic imaging system and are further classified by a customized pseudo-3D convolutional network. To evaluate the performance of our technique, we detected serially-diluted MCF7 cancer cells that were spiked in whole blood, achieving a limit of detection (LoD) of 10 cells per 1 mL of whole blood.
We report a single-shot computational polarized-light microscopy (SCPLM) method for identifying pathological crystals in bodily-fluids. We utilize the four-directional polarizers integrated on the pixels of a CMOS image sensor to reconstruct the transmittance, retardance, and slow-axis orientation maps of the objects with a single image exposure. Using SCPLM, we imaged birefringent crystals found in synovial fluid, e.g., monosodium urate, calcium pyrophosphate dehydrate, and triamcinolone acetonide. The quantitative birefringence images created by our method are pseudo-colored and digitally-integrated with bright-field images to highlight the birefringent crystals within the background. We believe this single-shot, quantitative, and easy-to-operate method will significantly benefit rheumatology.
We report a highly-sensitive, high-throughput, and cost-effective bacteria identification system which continuously captures and reconstructs holographic images of an agar-plate and analyzes the time-lapsed images with deep learning models for early detection of colonies. The performance of our system was confirmed by detection and classification of Escherichia coli, Enterobacter aerogenes, and Klebsiella pneumoniae in water samples. We detected 90% of the bacterial colonies and their growth within 7-10h (>95% within 12h) with ~100% precision, and correctly identified the corresponding species within 7.6-12h with 80% accuracy, and achieved time savings of >12h as compared to the gold-standard EPA-approved methods.
Forest fires are a major source of particulate matter (PM) air pollution on a global scale. The composition and impact of PM are typically studied using only laboratory instruments and extrapolated to real fire events owing to a lack of analytical techniques suitable for field-settings. To address this and similar field test challenges, we developed a mobilemicroscopy- and machine-learning-based air quality monitoring platform called c-Air, which can perform air sampling and microscopic analysis of aerosols in an integrated portable device. We tested its performance for PM sizing and morphological analysis during a recent forest fire event in La Tuna Canyon Park by spatially mapping the PM. The result shows that with decreasing distance to the fire site, the PM concentration increases dramatically, especially for particles smaller than 2 µm. Image analysis from the c-Air portable device also shows that the increased PM is comparatively strongly absorbing and asymmetric, with an aspect ratio of 0.5–0.7. These PM features indicate that a major portion of the PM may be open-flame-combustion-generated element carbon soot-type particles. This initial small-scale experiment shows that c-Air has some potential for forest fire monitoring.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.