This PDF file contains the front matter associated with SPIE Proceedings Volume 12032, including the Title Page, Copyright information, Table of Contents, and Conference Committee listings.

With the increased availability of extended reality (XR) devices in the marketplace, there has been a rapid development of medical XR applications spanning from education, training, rehabilitation, pre-procedural planning, and intra-procedural use. We will explore various use case to understand the importance of technology-use case matches and focus on intra-procedural use cases which generally have the highest risk to patient and medical provider but may have the most sizable impact on benefit to patient and procedure.

Non-linear gradients impact diffusion weighted (DW) MRI by corrupting the experimental setup and lead to problems during image encoding including the effects in-plane distortion, in-plane shifts, intensity modulations and phase errors. Recent studies have been shown this may present significant complication in the interpretation of results and conclusion while studying tractography and tissue microstructure in data. To interpret the degree in consequences of gradient non-linearities between the desired and achieved gradients, we introduced empirically derived gradient non linear fields at different orientations and different tensor properties. The impact is assessed through diffusion tensor properties including mean diffusivity (MD), fractional anisotropy (FA) and principal eigen vector (PEV). The study shows lower FA are more susceptible to LR fields and LR fields with determinant <1 or <1 corrupt tensor more. The corruption can result in significantly different FA based on true-FA and LR field. Apparent MD decreases for negative determinant, on the other hand positive determinant shows the opposite effect. LR field have a larger impact on PEV when FA value is small. The results are dependent on the underlying orientation, non-linear field corruption can cause both increase and decrease of estimated FA, MD and PEV value. This work provides insight into characterizing the non-linear gradient error and aid in selecting correction techniques to address the inaccuracies in b-values.

Inpainting shadowed regions cast by superficial blood vessels in retinal optical coherence tomography (OCT) images is critical for accurate and robust machine analysis and clinical diagnosis. Traditional sequence-based approaches such as propagating neighboring information to gradually fill in the missing regions are cost-effective. But they generate less satisfactory outcomes when dealing with larger missing regions and texture-rich structures. Emerging deep learning-based methods such as encoder-decoder networks have shown promising results in natural image inpainting tasks. However, they typically need a long computational time for network training in addition to the high demand on the size of datasets, which makes it difficult to be applied on often small medical datasets. To address these challenges, we propose a novel multi-scale shadow inpainting framework for OCT images by synergically applying sparse representation and deep learning: sparse representation is used to extract features from a small amount of training images for further inpainting and to regularize the image after the multi-scale image fusion, while convolutional neural network (CNN) is employed to enhance the image quality. During the image inpainting, we divide preprocessed input images into different branches based on the shadow width to harvest complementary information from different scales. Finally, a sparse representation-based regularizing module is designed to refine the generated contents after multi-scale feature aggregation. Experiments are conducted to compare our proposal versus both traditional and deep learning-based techniques on synthetic and real-world shadows. Results demonstrate that our proposed method achieves favorable image inpainting in terms of visual quality and quantitative metrics, especially when wide shadows are presented.

Accurate measurement of strain in a deforming organ has been an important step in motion analysis using medical images. In recent years, internal tissue’s in vivo motion and strain computation is mostly achieved through dynamic magnetic resonance (MR) imaging. However, such data lack information on tissue’s intrinsic fiber directions, preventing computed strain tensors from being projected onto a direction of interest. Although diffusion-weighted MR imaging excels at providing fiber tractography, it yields static images unmatched with dynamic MR data. In this work, we report an algorithm workflow that estimates strain values in the diffusion MR space by matching corresponding tagged dynamic MR images. We focus on processing a dataset of various human tongue deformations in speech. The geometry of tongue muscle fibers is provided by diffusion tractography, while spatiotemporal motion fields are provided by tagged MR analysis. The tongue’s deforming shapes are determined by segmenting a synthetic cine dynamic MR sequence generated from tagged data using a deep neural network. Estimated motion fields are transformed into the diffusion MR space using diffeomorphic registration, eventually leading to strain values computed in the direction of muscle fibers. The method was tested on 78 time volumes acquired during three sets of specific tongue deformations including both speech and protrusion motion. Strain in the line of action of seven internal tongue muscles was extracted and compared both intra- and inter-subject. Resulting compression and stretching patterns of individual muscles revealed unique behavior of individual muscles and their potential activation pattern.

Optical coherence tomography (OCT) is a prevalent non-invasive imaging method which provides high resolution volumetric visualization of retina. However, its inherent defect, the speckle noise, can seriously deteriorate the tissue visibility in OCT. Deep learning based approaches have been widely used for image restoration, but most of these require a noise-free reference image for supervision. In this study, we present a diffusion probabilistic model that is fully unsupervised to learn from noise instead of signal. A diffusion process is defined by adding a sequence of Gaussian noise to self-fused OCT b-scans. Then the reverse process of diffusion, modeled by a Markov chain, provides an adjustable level of denoising. Our experiment results demonstrate that our method can significantly improve the image quality with a simple working pipeline and a small amount of training data. The implementation is available at https://github.com/DeweiHu/OCT_DDPM.

Digital Breast Tomosynthesis (DBT) projections are acquired with a high level of noise, compared to Digital Mammography (DM) projections. Noise reduction in DBT projections is important because the projections are obtained with low radiation dose, elevating the noise level. In this way, noise reduction is essential to improve the quality of DBT exam. Recently, neural network based methods have been applied to denoise DBT projections, reaching remarkable results. Some papers have been published showing that these methods are able to overpass traditional methods’ results, but we could not find a paper comparing the different types of networks to denoise DBT projections. In this paper, we proposed an experiment to compare neural network based methods, with different architecture types, and traditional methods. We performed a comparison among five traditional non blind denoising methods and six neural network models. Considering both quantitative and qualitative analysis, we found that some neural network models achieve remarkable results, especially shallower models.

Choroid plexuses (CP) are structures of the brain ventricles which produce most of the cerebrospinal fluid (CSF). Several postmortem and in vivo studies have pointed towards their role in the inflammatory processes in multiple sclerosis (MS). Automatic segmentation of CP from MRI thus has high value for studying their characteristics in large cohorts of patients. To the best of our knowledge, the only freely available tool for CP segmentation is FreeSurfer but its accuracy for this specific structure is poor. In this paper, we propose to automatically segment CP from non-contrast enhanced T1-weighted MRI. To that end, we introduce a new model called ”Axial-MLP” based on an assembly of Axial multi-layer perceptrons (MLPs). This is inspired by recent works which showed that the self-attention layers of Transformers can be replaced with MLPs. This approach is systematically compared with a standard 3D U-Net, nnU-Net, Freesurfer and FastSurfer. For our experiments, we make use of a dataset of 141 subjects (44 controls and 97 patients with MS). We show that all the tested deep learning (DL) methods outperform FreeSurfer (Dice around 0.7 for DL vs 0.33 for FreeSurfer). Axial-MLP is competitive with U-Nets even though it is slightly less accurate. The conclusions of our paper are two-fold: 1) the studied deep learning methods could be useful tools to study CP in large cohorts of MS patients; 2) Axial-MLP is a potentially viable alternative to convolutional neural networks for such tasks, although it could benefit from further improvements. An implementation is available at https://github.com/aramis-lab/axial-mlp.

Radiotherapy plays an important role in the management of lung cancer for many patients. During radiation treatment the respiratory motion of the patient impedes the possibility to accurately target the tumor, requiring large treatment margins. This leads to additional radiation to healthy tissue, and the associated toxicity. Real time adaptive radiotherapy is a promising direction for solving this problem, where radiation beams are shaped continuously to track the tumor based on real-time image analysis, which is possible with e.g. MRI-guided radiotherapy. To assist in the MR-Linac planning process, we developed a U-Net based tumor tracking method that uses a double encoder structure to incorporate both 3D+t planning CT images and a 3D planning scan with corresponding segmentation. Our best model achieves 0.60 surface Dice and 92% recall.

Heart segmentation plays an important role in accurate diagnosis and treatment of cardiovascular disease. More recently, deep convolutional neural networks (CNN) are predominant to many medical image analysis applications including 3D heart segmentation. For example, 3D UNet with U-shape encoder-decoder architecture performs well in volumetric segmentation. However, standard convolution, the building block of 3D CNN network usually contains a large number of parameters. In this paper, our objective is to investigate a light convolution module to build an efficient network. Inspired by 2D Tied Block Convolution (TBC), we introduce a Tied Block 3D Convolution (TBC-3D) operator which reuses a small number of convolution filters across each channel group. To this end, TBC-3D requires fewer parameters and is able to obtain more feature maps with high performance. Furthermore, we combine TBC-3D with the Ghost-3D module to construct Ghost Tied Block (GTB). Specifically, Ghost module employs standard convolution (OP1) with few filters to obtain intrinsic feature maps, and then generates more features by cheap linear operation like depth-wise convolution (OP2). TBC-3D is applied in both OP1 and OP2 in the Ghost-3D module. Compared to state-of-the-art solutions using 3D UNet-like architecture, our model with GTB achieves competitive performance on the MM-WHS whole heart segmentation Challenge 2017 datasets with 2.31x less parameters and 1.93x fewer FLOPs.

One of the most promising approaches for unsupervised learning is combining deep representation learning and deep clustering. Recent studies propose to simultaneously learn representation using deep neural networks and perform clustering by defining a clustering loss on top of embedded features. Unsupervised image clustering naturally requires good feature representations to capture the distribution of the data and subsequently differentiate data points from one another. Among existing deep learning models, the generative variational autoencoder explicitly learns data generating distribution in a latent space. We propose a Deep Variational Clustering (DVC) framework for unsupervised representation learning and clustering of large-scale medical images. DVC simultaneously learns the multivariate Gaussian posterior through the probabilistic convolutional encoder, and the likelihood distribution with the probabilistic convolutional decoder; and optimizes cluster labels assignment. Here, the learned multivariate Gaussian posterior captures the latent distribution of a large set of unlabeled images. Then, we perform unsupervised clustering on top of the variational latent space using a clustering loss. In this approach, the probabilistic decoder helps to prevent the distortion of data points in the latent space, and to preserve local structure of data generating distribution. The training process can be considered as a self-training process to refine the latent space and simultaneously optimizing cluster assignments iteratively. We evaluated our proposed framework on three public datasets that represented different medical imaging modalities. Our experimental results show that our proposed framework generalizes better across different datasets. It achieves compelling results on several medical imaging benchmarks. Thus, our approach offers potential advantages over conventional deep unsupervised learning in real-world applications. The source code of the method and of all the experiments are available publicly at: https://github.com/csfarzin/DVC

Detection of lung diseases from chest X-rays has been of great interest from the research community during the last decade. Despite the existence of large annotated public databases, computer-aided diagnostic solutions still fail on challenging rare abnormality cases. In this study, we investigated the paradigm of combining the analysis of chest X-rays and physician gaze patterns during the analysis of these X-rays to improve the computerized diagnostic accuracy. Tobii Eye Tracker 4C has been mounted to a physician workstation and his eye movements were recorded during the analysis of 400 chest X-rays in two days of work. The X-rays have been sampled from CheXpert, RSNA, and SIIM-ACR public databases labeled with 14 different pathology types. The task was formulated as a binary classification problem. A ResNet34-based neural network has been trained to map the input chest X-ray with the output physician gaze map and binary pathology label. The proposed network improved the diagnostic accuracy to 0.714 of the area under receiving operator curve (AUC) from 0.681 AUC obtained for the same ResNet34 trained to generate binary pathology labels alone. The proposed study has demonstrated the potential benefits of using gaze information in computerized diagnostic solutions.

Segmenting medical images accurately and reliably is important for disease diagnosis and treatment. It is a challenging task because of the wide variety of objects’ sizes, shapes, and scanning modalities. Recently, many convolutional neural networks (CNN) have been designed for segmentation tasks and achieved great success. Few studies, however, have fully considered the sizes of objects, and thus most demonstrate poor performance for small objects segmentation. This can have a significant impact on the early detection of diseases. This paper proposes a Context Axial Reserve Attention Network (CaraNet) to improve the segmentation performance on small objects compared with several recent state-of-the-art models. We test our CaraNet on brain tumor (BraTS 2018) and polyp (Kvasir-SEG, CVC-ColonDB, CVC-ClinicDB, CVC-300, and ETIS-LaribPolypDB) segmentation datasets. Our CaraNet achieves the top-rank mean Dice segmentation accuracy, and results show a distinct advantage of CaraNet in the segmentation of small medical objects. Codes available: https://github.com/AngeLouCN/CaraNet

Deep learning promises the extraction of valuable information from traumatic brain injury (TBI) datasets and depends on efficient navigation when using large-scale mixed computed tomography (CT) datasets from clinical systems. To ensure a cleaner signal while training deep learning models, removal of computed tomography angiography (CTA) and scans with streaking artifacts is sensible. On massive datasets of heterogeneously sized scans, time-consuming manual quality assurance (QA) by visual inspection is still often necessary, despite the expectation of CTA annotation (artifact annotation is not expected). We propose an automatic QA approach for retrieving CT scans without artifacts by representing 3D scans as 2D axial slice montages and using a multi-headed convolutional neural network to detect CT vs CTA and artifact vs no artifact. We sampled 848 scans from a mixed CT dataset of TBI patients and performed 4-fold stratified cross-validation on 698 montages followed by an ablation experiment—150 stratified montages were withheld for external validation evaluation. Aggregate AUC for our main model was 0.978 for CT detection, 0.675 for artifact detection during cross validation and 0.965 for CT detection, 0.698 for artifact detection on the external validation set, while the ablated model showed 0.946 for CT detection, 0.735 for artifact detection during cross-validation and 0.937 for CT detection, 0.708 for artifact detection on the external validation set. While our approach is successful for CT detection, artifact detection performance is potentially depressed due to the heterogeneity of present streaking artifacts and a suboptimal number of artifact scans in our training data.

Deep learning models for organ contouring in radiotherapy are poised for clinical usage, but currently, there exist few tools for automated quality assessment (QA) of the predicted contours. Bayesian models and their associated uncertainty, can potentially automate the process of detecting inaccurate predictions. We investigate two Bayesian models for auto-contouring, DropOut and FlipOut, using a quantitative measure – expected calibration error (ECE) and a qualitative measure – region-based accuracy-vs-uncertainty (R-AvU) graphs. It is well understood that a model should have low ECE to be considered trustworthy. However, in a QA context, a model should also have high uncertainty in inaccurate regions and low uncertainty in accurate regions. Such behaviour could direct visual attention of expert users to potentially inaccurate regions, leading to a speed-up in the QA process. Using R-AvU graphs, we qualitatively compare the behaviour of different models in accurate and inaccurate regions. Experiments are conducted on the MICCAI2015 Head and Neck Segmentation Challenge and on the DeepMindTCIA CT dataset using three models: DropOut-DICE, Dropout-CE (Cross Entropy) and FlipOut-CE. Quantitative results show that DropOut-DICE has the highest ECE, while Dropout-CE and FlipOut-CE have the lowest ECE. To better understand the difference between DropOut-CE and FlipOut-CE, we use the R-AvU graph which shows that FlipOut-CE has better uncertainty coverage in inaccurate regions than DropOut-CE. Such a combination of quantitative and qualitative metrics explores a new approach that helps to select which model can be deployed as a QA tool in clinical settings.

Fully automatic classification of magnetic resonance (MR) brain images into different contrasts is desirable for facilitating image processing pipelines, as well as for indexing and retrieving from medical image archives. In this paper, we present an approach based on a Siamese neural network to learn a discriminative feature representation for MR contrast classification. The proposed method is shown to outperform a traditional deep convolutional neural network method and a template matching method in identifying five different MR contrasts of input brain volumes with a variety of pathologies, achieving 98.59% accuracy. In addition, our approach permits one-shot learning, which allows generalization to new classes not seen in the training set with only one example of each new class. We demonstrate accurate one-shot learning performance on a sixth MR contrast that was not included in the original training.

Medical image segmentation is one of the core tasks of medical image analysis. Automatic segmentation of brain magnetic resonance images (MRIs) can be used to visualize and track changes of the brain’s anatomical structures that may occur due to normal aging or disease. Machine learning techniques are widely used in automatic structure segmentation. However, the contrast variation between the training and testing data makes it difficult for segmentation algorithms to generate consistent results. To address this problem, an image–to– image translation technique called MR image harmonization can be used to match the contrast between different data sets. It is important for the harmonization to transform image intensity while maintaining the underlying anatomy. In this paper, we present a 3D U-Net algorithm to segment the thalamus from multiple MR image modalities and investigate the impact of harmonization on the segmentation algorithm. Manual delineations of thalamic nuclei on two data sets are available. However, we aim to analyze the thalamus in another large data set where ground truth labels are lacking. We trained two segmentation networks, one with unharmonized images and the other with harmonized images, on one data set with manual labels, and compared their performances on the other data set with manual labels. These two data groups were diagnosed with two brain disorders and were acquired with similar imaging protocols. The harmonization target is the large data set without manual labels, which also has a different imaging protocol. The networks trained on unharmonized and harmonized data showed no significant difference when evaluating on the other data set; demonstrating that image harmonization can maintain the anatomy and does not affect the segmentation task. The two networks were evaluated on the harmonization target data set and the network trained on harmonized data showed significant improvement over the network trained on unharmonized data. Therefore, the network trained on harmonized data provides the potential to process large amounts of data from other sites, even in the absence of site-specific training data.

The development of automatic whole brain segmentation algorithms has greatly facilitated large-scale multi cohort magnetic resonance (MR) image analyses in recent years. However, the performance of these segmentation algorithms is often affected by image contrast due to the variations in pulse sequences, acquisitions parameters, and manufacturers. Quantitatively evaluating segmentation algorithms on different image contrasts is challenging because manual delineations of the human brain are usually limited. In this study, we tackle the problem by synthesizing new contrast MR images from a small set of images with manual delineations. We quantitatively evaluate the current state-of-the-art whole brain segmentation algorithm, SLANT, on various MR image contrasts. Based on 50 manually delineated T1-weighted MR images acquired from a single site, we synthesize new contrast images using a deep learning-based harmonization algorithm. Two types of contrast synthesis were conducted to simulate both intra- and inter-site contrast variability in MR imaging. SLANT performance is measured using the Dice similarity coefficient (DSC). Experiments show that the average DSC of SLANT varies with image contrast. We also demonstrate the preferred and the least preferred contrast of SLANT based on 11 real MR imaging sites.

This video: Explainable artificial-intelligence for diagnosis and risk estimation from cardiovascular images was recorded for Medical Imaging 2022.

Opportunistic screening of the spine using routine 3D-CT scans could be beneficial for early diagnosis and prevention of osteoporosis and degenerative diseases. In clinical practice, a software that alerts radiologists for signs of osteoporosis and degenerative deformities should be accurate and robust despite being limited by computational resources. We explore a light-weight alternative to existing vertebrae segmentation and labelling algorithms in our two-stage diagnosis pipeline and evaluate the efficiency and robustness of our proposed Deep Learning method. During our first stage, semantic segmentation and labelling of the vertebrae is performed using a low-complexity 3D-CNN. Our efficient architecture includes a MobileNetv2 backbone and DeepLab for segmentation. We optimise the network architecture to improve efficiency by applying the compound scaling idea of the EfficientNet. The first stage of our model is trained and evaluated on the public VerSe dataset, resulting in a multi-label Dice score of 75% while taking less than 0.05s inference time on GPU. The segmentation outcome can be further used to extract centre coordinates of each vertebra which are classified by a 3D-CNN into normal vertebra, degenerative deformities and osteoporotic fractures. Our preliminary results on the DiagnostikBilanz dataset, using centre coordinates, yield an F1-score of 76%. Our fully-automatic pipeline achieves an F1-Score of 74.6%, which is an improvement of 7% compared to the pipeline using nnUNet for segmentation in the first stage. Our method provides a light-weight solution to assist radiologists in differentiating osteoporotic fractures from degenerative deformities in opportunistic screening of the spine in CT scans. In particular, it allows to incorporate segmentation information into the challenging differentiation of degenerative deformities from osteoporotic fractures.

Deep learning has shown successful performance not only in supervised disease detection but also lesion localization under the weakly supervised learning framework with medical image processing. However, few consider the semantic relationship among the diseases and lesions which plays a critical role in actual clinical diagnosis. In this work, we propose a novel framework: Feature map Graph Representational Probabilistic Class Activation Map (FGR-PCAM) to learn the graph structure of lesion-specific features and consider these relationships while also leveraging the localization ability of PCAM. Considering the relations of localized lesion-specific features has been shown to enhance both thoracic diseases classification and localization tasks on CheXpert and Chest Xray14 datasets. Accurate classification and localization of Chest X-ray images would also help us fight against the COVID-19 and unveil COVID-19 fingerprints.

Age-related macular degeneration is the leading cause of vision deterioration among older adults. The selected treatment for this retinal disease is largely dependent on the later stage type (wet or geographic atrophy), meaning correct type classification is crucial to ensuring the best patient outcome. Previous studies have demonstrated high classification accuracy with medical images and used saliency maps to add explainability to opaque deep learning models. However, these explanations have revealed a tendency to make classification decisions based on irrelevant information. Our proposed deep learning model allows domain experts to correct model behavior during the training process through direct annotations of the regions of interest (ROIs) and integrates these annotations into the learning model. Our approach performs consistently with non-interactive classification accuracy of the retinal optical coherence tomography (OCT) scans. Filters are applied regionally to the original OCT image based on the annotations and Grad-CAM highlighted regions. Four interactive classification methods are introduced and compared against a non-interactive CNN with the same overall architecture. Three of the four methods selectively filter regions of the images with weighted pairs of enhancement and blurring filters. The fourth uses ROI maps to focus the attention of the feature maps on the expert annotated region(s). All overlap scores measuring the human and computer output agreement overperformed the non-interactive CNN baseline model with two of the interactive methods doubling the overlap score while another tripling the overlap score

In severe cases, placenta accreta spectrum (PAS) requires emergency hysterectomy, endangering the life of both mother and fetus. Early prediction may reduce complications and aid in management decisions in these high-risk pregnancies. In this work, we developed a novel convolutional network architecture to combine MRI volumes, radiomic features, and custom feature maps to predict PAS severe enough to result in hysterectomy after fetal delivery in pregnant women. We trained, optimized, and evaluated the networks using data from 241 patients, in groups of 157, 24, and 60 for training, validation, and testing, respectively. We found the network using all three paths produced the best performance, with an AUC of 87.8, accuracy 83.3%,sensitivity of 85.0, and specificity of 82.5. This deep learning algorithm, deployed in clinical settings, may identify women at risk before birth, resulting in improved patient outcomes.

There are many different sources of information beyond the actual images that serve to inform radiologists when they are making diagnoses. Therefore, it is important to include such information, referred to as context, in machine learning algorithms designed for automated classification. To better utilize and to explore the power of context in medical imaging we developed a classification algorithm based on convolutional neural networks (CNNs) with various contexts to classify liver lesions in multi-phase computed tomography data. We designed an algorithm, referred to as sliced 3D, to efficiently handle 3D context in the image data. We further included clinical context summarizing the patient’s medical status to improve performance, and we also exploited the presence or absence of other lesions in the volume to inform classification of a lesion under consideration. The effect of co-occurrence is learned from the data during the training process. The algorithm was developed, validated, and tested on 2205 multi-vendor multi-institution studies. We found that the sliced 3D algorithm performed better than equivalent 2D and 3D CNN baselines (average F1=0.629 vs F1=0.589). Using the clinical and co-occurrence contexts further improved the algorithm’s performance (average F1=0.734). Our evaluation demonstrates that this novel CNN architecture, in conjunction with additional information about medical status and lesion co-occurrence, substantially improves classification results.

Deep Neural Networks (DNN) have made rapid progress in medical image analysis, and it is now common to see algorithms proposed in the literature that claim to meet or exceed clinician performance. However, DNNs typically operate like black box systems. Deployment of these algorithms for safety-critical tasks, such as medical applications, is therefore challenging without methods to characterize and identify the robustness of DNNs in a generalized setting. Furthermore, DNNs are known to be sensitive to small changes in input data, and may rely on spurious correlations seen in the training samples which precludes them from generalizing to data with a different distribution. We previously proposed an attribute-ranking algorithm that ranks discrete data attributes by how informative they are about the DNN performance using Mutual Information. In this study we leverage this algorithm in a novel way to determine if the data attributes that impact the DNN performance are clinically relevant and thereby whether the DNN predictions are robust. We demonstrate the applicability of this method on melanoma classification with a DNN trained on the publicly available HAM 10,000 dataset and achieve 0.855 AUC on the held-out HAM test set. Our analysis identifies that image saturation, which is not a clinically relevant feature, is highly indicative of both whether an image is melanomic in the HAM training data and whether the DNN prediction is correct. Further testing reveals that when the classifier is tested on the SIIM-ISIC Melanoma dataset, where the correlation between image saturation and melanoma is not present, the classifier achieves only an AUC of 0.591, confirming that the DNN is not robust.

In this work, we propose a deep U-Net based model to tackle the challenging task of prostate cancer segmentation by aggressiveness in MRI based on weak scribble annotations. This model extends the size constraint loss proposed by Kervadec et al.¹ in the context of multiclass detection and segmentation task. This model is of high clinical interest as it allows training on prostate biopsy samples and avoids time-consuming full annotation process. Performance is assessed on a private dataset (219 patients) where the full ground truth is available as well as on the ProstateX-2 challenge database, where only biopsy results at different localisations serve as reference. We show that we can approach the fully-supervised baseline in grading the lesions by using only 6.35% of voxels for training. We report a lesion-wise Cohen’s kappa score of 0.29 ± 0.07 for the weak model versus 0.32 ± 0.05 for the baseline. We also report a kappa score (0.276 ± 0.037) on the ProstateX-2 challenge dataset with our weak U-Net trained on a combination of ProstateX-2 and our dataset, which is the highest reported value on this challenge dataset for a segmentation task to our knowledge.

Neural network-based video segmentation has proven effective in producing temporally-coherent segmentation and motion tracking of heart substructures in echocardiography. However, prior methods confine analysis to half-heartbeat systolic phase clips from end-diastole (ED) to end-systole (ES), requiring the specification of these frames in the video and limiting clinical applicability. Here we introduce CLAS-FV, a fully automated framework that extends upon this prior work, providing joint semantic segmentation and motion tracking in multi-beat echocardiograms. Our framework first employs a modified R2+1D ResNet stem, which is efficient in encoding spatiotemporal features, and further leverages sliding windows for both training and test time augmentation to accommodate the full cardiac cycle. First, through 10-fold cross-validation on the half-beat CAMUS dataset, we show that the R2+1D-based stem outperforms the prior 3D U-Net both in Dice overlap for all substructures, and in derived clinical indices of ED and ES ventricular volumes and ejection fraction (EF). Next, we use the large clinical EchoNet-Dynamic dataset to extend our framework to full multi-beat video segmentation. We obtain mean Dice overlap of 0.94/0.91 on left ventricle endocardium in ED/ES phases, and accurately infer EF (mean absolute error 5.3%) over 1269 test patients. The presented multi-heartbeat video segmentation framework promises fast and coherent segmentation and motion tracking for the rich phenotypic analysis of echocardiography.

Deep Neural Networks (DNNs) have the potential for making various clinical procedures more time-efficient by automating medical image segmentation. Due to their strong, in some cases human-level, performance, they have become the standard approach in this field. The design of the best possible medical image segmentation DNNs, however, is task-specific. Neural Architecture Search (NAS), i.e., the automation of neural network design, has been shown to have the capability to outperform manually designed networks for various tasks. However, the existing NAS methods for medical image segmentation have explored a quite limited range of types of DNN architectures that can be discovered. In this work, we propose a novel NAS search space for medical image segmentation networks. This search space combines the strength of a generalised encoder-decoder structure, well known from U-Net, with network blocks that have proven to have a strong performance in image classification tasks. The search is performed by looking for the best topology of multiple cells simultaneously with the configuration of each cell within, allowing for interactions between topology and cell-level attributes. From experiments on two publicly available datasets, we find that the networks discovered by our proposed NAS method have better performance than well-known handcrafted segmentation networks, and outperform networks found with other NAS approaches that perform only topology search, and topology-level search followed by cell-level search.

The purpose of this paper is to introduce a practical framework of using proxy data in automatic hyperparameter optimization for 3D multi-organ segmentation. The automated segmentation of abdominal organs from CT volumes is a main task in the medical image analysis field. Much research has been investigated to handle this task based on the immense experience of machine learning. Deep learning approaches require enormous experiments to design the optimal configurations for the best performance. Automatic machine learning (AutoML) using hyperparameter optimization to search the optimal training strategy makes it possible to find the appropriate settings without much deep experience. However, biases of training data can be highly related to the AutoML performance and efficiency. In this paper, we propose an AutoML framework that uses pre-selected proxy data to represent the entire dataset which has the potential to reduce the computation time needed for efficient hyperparameter optimization in searching learning. Both quantitative and qualitative results showed that our framework can effectively build more powerful segmentation models than manually designed deep-learning-based methods and AutoML, which use carefully tuned hyperparameters and randomly selected training subsets, respectively. The average Dice score for 10-class abdominal organ segmentation was 85.9%.

Automatically segmenting bleeding regions is an essential task for computer-assisted surgery systems. Recent development in deep learning has boosted the performance of medical image segmentation. However, training deep neural networks, in general, requires high-quality pixel-wise annotations, in which such an annotating process is expensive and easy to introduce annotation noise. To address this issue, motivated by the observation that the noise should be data-dependent, we propose an uncertainty-guided label smoothing method instead of using a fixed label smoothing strategy. Aleatoric uncertainty, which accounts for inherent noise such as annotation error, is estimated via an additional branch of deep neural networks. With the help of estimated aleatoric uncertainty, we could guide the spatial label smoothing in a self-adaptive manner. We demonstrated the effectiveness of our proposal by evaluating the Dice coefficient in a private bleeding segmentation dataset. An improvement over the baseline was observed for our proposal.

Segmentation of lymphoma lesions is challenging due to their varied sizes and locations in whole-body PET scans. In this work, we present a fully-automated segmentation technique using a multi-center dataset of diffuse large B-cell lymphoma (DLBCL) with heterogeneous characteristics. We utilized a dataset of [18F]FDG-PET scans (n=194) from two different imaging centers including cases with primary mediastinal large B-cell lymphoma (PMBCL) (n=104). Automated brain and bladder removal approaches were utilized as preprocessing steps, to tackle false positives caused by normal hypermetabolic uptake in these organs. Our segmentation model is a convolutional neural network (CNN), based on a 3D U-Net architecture that includes squeeze and excitation (SE) modules. Hybrid distribution, region, and boundary-based losses (Unified Focal and Mumford-Shah (MS)) were utilized that showed the best performance compared to other combinations (p<0.05). Cross-validation between different centers, DLBCL and PMBCL cases, and three random splits were applied on train/validation data. The ensemble of these six models achieved a Dice similarity coefficient (DSC) of 0.77 ± 0.08 and Hausdorff distance (HD) of 16.5 ±12.5. Our 3D U-net model with SE modules for segmentation with hybrid loss performed significantly better (p<0.05) as compared to the 3D U-Net (without SE modules) using the same loss function (Unified Focal and MS loss) (DSC= 0.64 ± 0.21 and HD= 26.3 ± 18.7). Our model can facilitate a fully automated quantification pipeline in a multi-center context that opens the possibility for routine reporting of total metabolic tumor volume (TMTV) and other metrics shown useful for the management of lymphoma.

Glaucoma is one of the leading causes of permanent blindness due to optic nerve damage. Optical coherence tomography (OCT) has become an important clinical tool for assessing structural damage from the loss of neurons. Traditional 2D and 3D methods have been successfully applied to quantify inner retinal layer thickness. However, these methods show less reliable segmentation in severe glaucoma when the retinal layers have become thin and violate algorithm assumptions. Deep learning (DL) is an alternative image analysis approach due to its powerful ability to extract features directly from data. State-of-the-art DL segmentation approaches can achieve sub-pixel accuracy at multiple retinal surfaces in OCT scans from normal eyes. However, limitations, such as spike-like segmentation errors (showing as high Hausdorff distances) and lack of contextual information from the input image, still need to be improved. To address these limitations, three novel solutions were proposed in this study. First, for data augmentation, we reconstructed more B-scans by reassembling A-scans at the vertical and jittered planes to expose DL to a greater variety of features encountered in OCT. Second, smoothed and contrast-enhanced images of each three adjacent B-scans were concatenated to provide a six-channel input image stack to the neural network with contextual information. Finally, we merged the predicted surfaces from both horizontal and vertical B-scans while maintaining retinal topological order. In our independently tested dataset, which included eyes with severe glaucoma, the proposed approach outperformed the state-of-the-art methods in mean absolute surface distances, Dice coefficients, and Hausdorff distance at multiple surfaces.

Landmark detection is a critical component of the image processing pipeline for automated aortic size measurements. Given that the thoracic aorta has a relatively conserved topology across the population and that a human annotator with minimal training can estimate the location of unseen landmarks from limited examples, we proposed an auxiliary learning task to learn the implicit topology of aortic landmarks through a CNN-based network. Specifically, we created a network to predict the location of missing landmarks from the visible ones by minimizing the Implicit Topology loss in an end-to-end manner. The proposed learning task can be easily adapted and combined with Unet-style backbones. To validate our method, we utilized a dataset consisting of 207 CTAs, labeling four landmarks on each aorta. Our method outperforms the state-of-the-art Unet-style architectures (ResUnet, UnetR) in terms of localization accuracy, with only a light (#params=0.4M) overhead. We also demonstrate our approach in two clinically meaningful applications: aortic sub-region division and automatic centerline generation.

Carotid artery vessel wall thickness measurement is an essential step in the monitoring of patients with atherosclerosis. This requires accurate segmentation of the vessel wall, i.e., the region between an artery’s lumen and outer wall, in black-blood magnetic resonance (MR) images. Commonly used convolutional neural networks (CNNs) for semantic segmentation are suboptimal for this task as their use does not guarantee a contiguous ring-shaped segmentation. Instead, in this work, we cast vessel wall segmentation as a multi-task regression problem in a polar coordinate system. For each carotid artery in each axial image slice, we aim to simultaneously find two non-intersecting nested contours that together delineate the vessel wall. CNNs applied to this problem enable an inductive bias that guarantees ring-shaped vessel walls. Moreover, we identify a problem-specific training data augmentation technique that substantially affects segmentation performance. We apply our method to segmentation of the internal and external carotid artery wall, and achieve top-ranking quantitative results in a public challenge, i.e., a median Dice similarity coefficient of 0.813 for the vessel wall and median Hausdorff distances of 0.552 mm and 0.776 mm for lumen and outer wall, respectively. Moreover, we show how the method improves over a conventional semantic segmentation approach. These results show that it is feasible to automatically obtain anatomically plausible segmentations of the carotid vessel wall with high accuracy.

Despite the widespread use of Convolutional Neural Networks (CNNs) for segmentation in medical imaging, there is yet to be a validated method to determine what regions of input images inform the models’ decisions. To advance the field, we have 1) modified three general, prevalent methods of classification-attribution to be applicable for use with segmentation models, 2) developed a novel method of attribution explicitly for segmentation models, and 3) formulated validation metrics for these attributions so results can be quantitatively compared. To adapt existing methods of classification-attribution, we newly employed a weighted sum across attribution maps from each post-bottleneck layer. For our novel method of attribution (Kernel-Weighted Contribution), a weighted sum of activations from each kernel across all post-bottleneck layers was weighted by each kernel’s dependent and independent contributions to the segmentation. We used the generated attribution maps to mask the input images and generate new predicted segmentations. The methods were then scored based on their sensitivity to region importance (Prediction Preserved) and ability to only attribute relevant regions (Image Preserved). All three adapted classification-attribution methods showed a significant increase in both Prediction Preserved and Image Preserved scores. Kernel-Weighted Contribution showed a median decreased Image Preserved score of 2-12% and increased Prediction Preserved score of 12-21% compared with modified classification attribution methods. These new methods provide insight into how segmentation models use regions of input images. Clinically relevant features can consequently be extracted from both foreground and relevant background regions. Additionally, the metrics of validation facilitate a quantitative and objective comparison of segmentation-attribution methods.

Unsupervised domain adaptation (UDA) has been widely used to transfer knowledge from a labeled source domain to an unlabeled target domain to counter the difficulty of labeling in a new domain. The training of conventional solutions usually relies on the existence of both source and target domain data. However, privacy of the large-scale and well-labeled data in the source domain and trained model parameters can become the major concern of cross center/domain collaborations. In this work, to address this, we propose a practical solution to UDA for segmentation with a black-box segmentation model trained in the source domain only, rather than original source data or a white-box source model. Specifically, we resort to a knowledge distillation scheme with exponential mixup decay (EMD) to gradually learn target-specific representations. In addition, unsupervised entropy minimization is further applied to regularization of the target domain confidence. We evaluated our framework on the BraTS 2018 database, achieving performance on par with white-box source model adaptation approaches.

The communication of reliable uncertainty estimates is crucial in the effort towards increasing trust in Deep Learning applications for medical image analysis. Importantly, reliable uncertainty estimates should remain stable under naturally occurring domain shifts. In this study, we evaluate the relationship between epistemic uncertainty and segmentation quality under domain shift within two clinical contexts: optic disc segmentation in retinal photographs and brain tumor segmentation from multi-modal brain MRI. Specifically, we assess the behavior of two epistemic uncertainty metrics derived from i, a single UNet’s sigmoid predictions, ii, deep ensembles, and iii, Monte Carlo dropout UNets, each trained with both soft Dice and weighted cross-entropy loss. Domain shifts were modeled by excluding a group with a known characteristic (glaucoma for optic disc segmentation and low-grade glioma for brain tumor segmentation) from model development and using the excluded data as additional, domain-shifted test data. While the performance of all models dropped slightly on the domain-shifted test data compared to the in-domain test set, there was no change in the Pearson correlation coefficient between the uncertainty metrics and the Dice scores of the segmentations. However, we did observe differences in the performance of two quality assessment applications based on epistemic uncertainty between the segmentation tasks. We introduce a new metric, the empirical strength distribution, to better describe the strength of the relationship between segmentation performance and epistemic uncertainty on a dataset level. We found that failures of the studied quality assessment applications were largely caused by shifts in the empirical strength distributions between training, in-domain, and domain-shifted test datasets. In conclusion, quality assessment tools based on the strong relationship between epistemic uncertainty and segmentation quality can be stable under small domain shifts. Developers should thoroughly evaluate the strength relationships for all available data and, if possible, under domain shift to ensure the validity of these uncertainty estimates on unseen data.

Statistical shape models (SSMs) represent the distribution of labeled points across a training set of shapes. The standard practice for SSMs based on principal component analysis (PCA) is to use clipping, thresholding the latent representation so that all shapes lie within 3 standard deviations of the mean. This practice precludes the representation of shapes that are not well represented by the training set, constraining the model to realistic solutions, but making it impossible to work with shapes at the edges of the statistical population. In this study, we investigate the impact of clipping in a PCA-based SSM and whether using L2 regularization is a good replacement for clipping in the context of the automatic 2D to 3D reconstruction of the spine. We first show that using L2 regularization is equivalent to using a probabilistic PCA with two error variables, accounting for the suppression of the least important principal components and for the fact that the training set cannot perfectly represent all shapes at test time. Secondly, we use two data sets of 1746 and 768 patients with adolescent idiopathic scoliosis to study the effect of regularization, for different regularization weights and with or without clipping, for removing landmark detection errors using a simulated noise or a reconstruction pipeline. In both sets of experiments, we show that regularization removes noise in a way similar to clipping without preventing the reconstruction of out-of-distribution shapes, leading to outputs closer to ground truth, demonstrating that using a regularized SSM should be preferred to clipping.

Pediatric spinal cord morphometry has been relatively understudied because of non-optimal image quality due to the difficulty of spine imaging, rarity of post-mortem analysis, motion artifacts, and pediatric MR imaging research focus on understanding spinal injury or pathology. The pediatric brain has been comparatively well-studied with white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF) differences observed with age and gender. Therefore, a greater understanding of pediatric cervical and thoracic spinal cord morphometry would be beneficial for developing clinically relevant cord growth models. We focused on retrospectively characterizing cervical and thoracic spinal cord growth and morphometry changes in a healthy pediatric population. High resolution multi-echo gradient echo (mFFE) images were acquired from pediatric spinal cord scans from 63 patients (mean: 9.24 years, range: 0.83-17.67 years). The mFFE scans were then registered to the template space for uniform viewing and analysis by using a customized semi-automatic processing pipeline involving Spinal Cord Toolbox (SCT). Jacobian control determinants were calculated, and subsequent WM, GM, dorsal column, lateral funiculi, and ventral funiculi scalar averaging was conducted. Random effects models were used to model age-related Jacobian scalar differences. Observing the growth of cord matter by patient age and vertebral level suggests that the upper cervical spinal cord, specifically C2-C3, and mid-thoracic spinal cord, T3-T8, grow faster than other cervical levels and thoracic levels, respectively. This knowledge will facilitate clinical decision making when considering spine interventions and conducting radiological analysis in children with cervical and thoracic spine abnormalities.

While state-of-the-art deep learning methods consistently provide the most accurate image segmentation results when sufficient training datasets are available, in some applications large datasets are difficult to acquire. For example, a model for inner ear structures can only be constructed by ex vivo specimen imaging modalities such as µCT. Constructing such datasets is costly and time consuming. Active shape models (ASM) have been a successful technique in medical image segmentation and require less extensive datasets for training. However, the ability of the ASM framework to capture complex shape variation is limited by representing variations across all global-pose-normalized training exemplars in a single, linear vector space. In this work, we describe a novel non-linear extension to the ASM in the form of a multi element ASMs. Instead of modelling shape from a single global pose as with the original ASM formulation, we capture differences in regional pose using a concept of multiple weighted ASMs which we call elements. Each element uses a unique set of landmark importance weights for use during the shape registration and model fitting process. Landmark weights are optimized to minimize the overall multi-element ASM’s fitting error on the training set shapes. We demonstrate the advantage of this approach in segmenting the labyrinth structure in the inner ear. We find that the multi element ASM consistently outperforms a traditional ASM on similar sample sizes, and multi-element ASMs trained on 10 samples outperform traditional ASMs trained with 15 samples. These results show the method’s potential advantages in applications that are limited by small shape libraries.

The cranial meninges are membranes enveloping the brain. The space between these membranes contains mainly cerebrospinal fluid. It is of interest to study how the volumes of this space change with respect to normal aging. In this work, we propose to combine convolutional neural networks (CNNs) with nested topology-preserving geometric deformable models (NTGDMs) to reconstruct meningeal surfaces from magnetic resonance (MR) images. We first use CNNs to predict implicit representations of these surfaces then refine them with NTGDMs to achieve sub-voxel accuracy while maintaining spherical topology and the correct anatomical ordering. MR contrast harmonization is used to match the contrasts between training and testing images. We applied our algorithm to a subset of healthy subjects from the Baltimore Longitudinal Study of Aging for demonstration purposes and conducted longitudinal statistical analysis of the intracranial volume (ICV) and subarachnoid space (SAS) volume. We found a statistically significant decrease in the ICV and an increase in the SAS volume with respect to normal aging.

Accurate segmentation of prostate magnetic resonance (MR) images provides sufficient information for the diagnosis and treatment of prostate cancer. However, the automatic segmentation of prostate based on 3D MR images faces some challenges. First, the prostate differs from other anatomical structures in that its gland lacks clear boundaries. Moreover, there are large differences in the background texture, shape, and size of different samples of prostate MR images, which makes segmentation accuracy difficult to improve. To address this issue, we propose an automatic segmentation model called the Shape Constraint U-Net (SCU-Net) for prostate MR images. The network focuses on the segmentation of the prostate boundary region by introducing a shape constraint stream based on a parallel encoder–decoder structure that can handle different shape and texture information. Specifically, the shape constraint stream is composed of a multi-level boundary attention module that can process boundary-related information using advanced activation features in the regular segmentation stream. Finally, the multi-scale context information extracted from the regular segmentation stream is fused with the multi-level boundary information obtained from the shape constraint stream to generate a segmentation prediction map. Owing to the additional shape constraints, the network substantially improves prostate boundary region segmentation. We evaluated our method on two different real clinical MR prostate datasets. The experimental results demonstrate that SCU-Net achieves state-of-the-art prostate segmentation accuracy, especially in boundary-region voxel prediction. Further analysis indicates that the proposed shape constraint stream can be used to improve the boundary voxel prediction performance of other segmentation networks.

Automated labeling of intracranial arteries facilitates the identification of risk predictors for aneurysm development, as well as the detection of aneurysms. Although such methods have been previously developed, accurate labeling is challenged by the large variation found in the configuration of the circulatory anastomosis of intracranial arteries, named the Circle of Willis (CoW). Recent studies have shown the potential of deep learning techniques with geometric features to handle large topological variation. We propose a method to incorporate node features based on an atlas in a graph convolutional net (GCN). Time-of-flight magnetic resonance angiography (MRAs) images without intracranial aneurysms were divided in a training set (N=32) and test set (N=16). The atlas was used to identify the coordinates of eleven main CoW artery bifurcations. Node features were obtained by computing the reciprocal Euclidean distances between these coordinates and each node position. Results showed statistically significant improvements of node classification using the atlas compared to other commonly used node features (p<0.005, Wilcoxon). We achieved an average recall of 0.84, precision of 0.71, and F1-score of 0.77 for all CoW nodes, with 4.1 ± 1.9 falsely classified nodes per image. The results indicate that atlas-based features boost the ability of a GCN to handle anatomical variants and smaller arteries. The performance of the model could be further improved by including additional MRAs. In addition, the model should be tested on MRAs from other institutions to assess reproducibility and generalizability.

Precise and robust multimodal image registration helps in spatially aligning medical finding from multiple sources for accurate medical diagnosis. We present MedRegNet, a lightweight descriptor module for registration multimodal retinal images. We utilize generative adversary networks (GANs) to learn generator networks which during training can synthesize structurally consistent multimodal image-pairs. From these image-pairs MedRegNet learns to predict stable point descriptors via a relaxed ranking loss. As no dataset-specific incentives are given, MedRegNet learns stable representations by itself. Most importantly, both mono- and multimodal training – including training the GANs – is entirely unsupervised, hence no costly expert annotation is required. Through evaluation on the publicly available Fundus Image Registration Dataset (FIRE) as well as on our own multimodal dataset containing 340 retinal fundus, autofluorescence and fluorescein angiography image-pairs from 24 patients we show MedRegNet to improve robustness and registration capabilities of classical detector/descriptor algorithms like SIFT, ORB, KAZE and AKAZE. Despite using the same interest points, MedRegNet is matching more points successfully than either baseline even in the monomodal use case. In the multimodal case, where the classical baselines fail due to large visual difference between the modalities, MedRegNet’s registration performance stays consistent to the monomodal case. Furthermore, MedRegNet shows to be adaptable to point detectors it was not trained on with no or only little cost in performance. MedRegNet can easily be integrated into any feature-based registration pipeline and due to the lack of dataset-specific incentives has the potential to be applied to fields outside retinal imaging.

Optical coherence tomography (OCT) retinal volumes are prone to motion artifacts due to the movement of the eye during acquisition. Current retrospective motion correction algorithms are either computationally expensive or limited to pair-wise formulations, based on registration of consecutive slices (B-scans). This type of approach can lead to errors when individual B-scans contain artifacts or lack sufficient signal. Instead, we propose a framework, based on unsupervised deep learning, that corrects motion by aligning groups of consecutive B-scans. The network architecture is fully-convolutional and, thus, it can perform inference on the entire OCT volume, even though it was trained on groups of smaller size. Moreover, we improved performance by inferring in a multi-shot recurrent manner, which was further leveraged by a novel data augmentation technique. We used an exhaustive search algorithm (brute-force) to compare the proposed method against, both quantitatively and qualitatively based on visual assessment. In a dataset of 146 (training: 106, validation: 40) macula and optic disc volumes from 19 healthy subjects, our best performing configuration achieved 72% reduction in registration errors compared to the exhaustive search algorithm, with a computation time of 2.35 seconds. These results demonstrated that our framework has the potential to provide a fast and robust solution, based on deep learning registration, for the motion correction of OCT images.

A novel technology for estimating both the pose and the joint flexion from a single musculoskeletal X-ray image is presented for automatic quality assessment of patient positioning. The method is based on convolutional neural networks and does not require pose or flexion labels of the X-ray images for the training phase. The task is split into two steps: (i) detection of relevant bone contours in the X-ray by a feature-detection network and (ii) regression of the pose and flexion parameters by a pose-estimation network based upon the detected contours. This separation enables the pose-estimation network to be trained using synthetic contours, which are generated via projections of an articulated 3D model of the target anatomy. It is demonstrated that the use of data-augmentation techniques during training of the pose-estimation network significantly contributes to the robustness of the algorithm. Feasibility of the approach is illustrated using lateral ankle X-ray exams. Validation was performed using X-rays of an anthropomorphic phantom of the foot-ankle joint, imaged in various controlled positions. Reference pose parameters were established by an expert using an interactive tool to align the articulated 3D joint model with the phantom image. Errors in pose estimation are in the range of 2 degrees per pose angle and at the level of the expert performance. Using the rigid foot phantom the flexion parameter was constant, but the overall results indicate accurate estimation also of this parameter.

Registration of microscopy images is an important task in biomedical applications. We introduce a deep learning approach for non-rigid registration of cell nuclei in temporal microscopy image sequences. First, we present a segmentation-based registration method which combines different transformation models and can cope with strong image intensity changes. Second, as an extension, we propose a joint segmentation and registration method which includes a cycle consistency loss and automatically determines the segmentation. Both methods do not need labeled data for network training. The methods were applied to live cell microscopy images of cell nuclei and yield better results than baseline methods.

Computed tomography (CT) scans have been widely used to evaluate lung health because of their ability to differentiate tissue densities. Recent studies such as COPDGene have collected inspiration and expiration CT scans from thousands of subjects, promising insight into the mechanical properties of lung tissue. These paired scans must be spatially aligned (i.e., registered) to extract biomarkers describing movement of lung tissue that may correlate with disease. Unfortunately, the relationship between registration and biomarker error is poorly characterized, a challenge to be addressed before registration-based biomarkers are used in clinical practice. In our analysis, we considered three registration-based biomarkers (Jacobian determinant, anisotropic deformation index, and slab-rod index) and demonstrated their sensitivity to modeled registration error. We provide a range of errors for a given biomarker, highlighting how both the magnitude of registration error and correlations between vectors in the registration error field influence biomarker error. We then describe a method to measure the error field for a particular registration algorithm and compare it with modeled registration error. These estimates enable the selection of an appropriate registration error model, which improve understanding of biomarker uncertainty. Quantifying the relationship between registration and biomarker error is crucial because it can inform registration algorithm selection to reduce error in future research studies, and in turn, result in robust imaging biomarkers for disease characterization.

Brain extraction, also known as skull stripping, from magnetic resonance images (MRIs) is an essential preprocessing step for many medical image analysis tasks and is also useful as a stand-alone task for estimating the total brain volume. Currently, many proposed methods have excellent performance on T1-weighted images, especially for healthy adults. However, such methods do not always generalize well to more challenging datasets such as pediatric, severely pathological, or heterogeneous data. In this paper, we propose an automatic deep learning framework for brain extraction on T1-weighted MRIs of adult healthy controls, Huntington’s disease patients and pediatric Aicardi Gouti`eres Syndrome (AGS) patients. We examine our method on the PREDICT-HD and the AGS datasets, which are multi-site datasets with different protocols/scanners. Compared to current state-of-the-art methods, our method produced the best segmentations with the highest Dice score, lowest average surface distance and lowest 95-percent Hausdorff distance on both datasets. These results indicate that our method has better accuracy and generalizability for heterogeneous T1-w MRI datasets.

Frontotemporal dementia (FTD) is a rare neurodegenerative disease, often of genetic origin, with no effective treatment. There is a substantial pathophysiological overlap with amyotrophic lateral sclerosis (ALS), mutations in the C9orf72 gene being their most common genetic cause. In these disorders, no single biomarker can accurately measure progression, thus it is crucial to combine complementary information from multiple modalities to evaluate new therapeutic interventions. In particular, neuroimaging and transcriptomic (microRNA) data have been shown to have value to track FTD and ALS progression. As these conditions are rare, large samples are not available, hence the need for methods to fuse multimodal data from small samples. In this paper, we propose a method for computing a disease progression score (DPS) from cross-sectional multimodal data, based on variational autoencoders (VAE). We show that unsupervised training leads to the estimation of meaningful latent spaces, where subjects with similar disease states are clustered together and from which a DPS may be inferred. Models were evaluated on 14 patients, 40 presymptomatic mutation carriers and 37 healthy controls from the PREV-DEMALS study. Since there is no ground truth for the DPS, we used the inferred scores to perform pairwise classification as a proxy metric. Presymptomatic subjects and patients were classified with an average area under the ROC curve of 0.83 and 0.94, respectively without and with feature selection. The proposed approach has the potential to leverage cross-sectional multimodal datasets with small sample sizes in order to objectively measure disease progression.

Magnetic resonance imaging of mice and other small animals plays an important role in translational medicine, where animal models are essential in the study of diseases and their potential treatments. While a large number of brain imaging studies using mice are conducted every year, there are few tools designed specifically for analyzing mouse MRI. Researchers often resort to adapting tools designed for processing MRI of human brains to work for the different structure, composition, and appearance of the mouse brain. While these methods may provide a reasonable initialization, researchers often have to perform extensive manual editing. In this work, we adapted a patch-based 2-Stage CNN Architecture to segment brain and non-brain in mouse MRI. We trained our model using brain MRI of healthy mice and mice with experimental autoimmune encephalomyelitis. These images had been previously acquired by our research team and processed to extract the brain using detailed manual editing. We compared our method with five existing tools, including BSE,¹ rBET,² AFNI,³ PCNN,⁴ and nnU-Net,⁵ using manually delineated mouse MRI. Both our method and nnU-Net achieved mean Dice scores on the order of 0.99 and HD95 measures on the order of one voxel, substantially outperforming the other methods. Our proposed method had slightly better Dice, HD99, and sensitivity measures than nnU-Net, while nnU-Net had slightly better HD95 scores. While these differences were small enough that further evaluation of the methods on a broader set of images would be warranted, they do suggest that our proposed method is competitive with a state-of-the-art deep learning method.

7T MRI provides unprecedented resolution for examining human brain anatomy in vivo. For example, 7T MRI enables deep thickness measurement of laminar subdivisions in the right fusiform area. Existing laminar thickness measurement on 7T is labor intensive, and error prone since the visual inspection of the image is typically along one of the three orthogonal planes (axial, coronal, or sagittal view). To overcome this, we propose a new analytics tool that allows flexible quantification of cortical thickness on a 2D plane that is orthogonal to the cortical surface (beyond axial, coronal, and sagittal views) based on the 3D computational surface reconstruction. The proposed method further distinguishes high quality 2D planes and the low-quality ones by automatically inspecting the angles between the surface normals and slice direction. In our approach, we acquired a pair of 3T and 7T scans (same subject). We extracted the brain surfaces from the 3T scan using MaCRUISE and projected the surface to the 7T scan’s space. After computing the angles between the surface normals and axial direction vector, we found that 18.58% of surface points were angled at more than 80 with the axial direction vector and had 2D axial planes with visually distinguishable cortical layers. 15.12% of the surface points with normal vectors angled at 30 or lesser with the axial direction, had poor 2D axial slices for visual inspection of the cortical layers. This effort promises to dramatically extend the area of cortex that can be quantified with ultra-high resolution in-plane imaging methods.

Characterizing relationships between gray matter (GM) and white matter (WM) in early Alzheimer's disease (AD) would improve understanding of how and when AD impacts the brain. However, modeling these relationships across brain regions and longitudinally remains a challenge. Thus, we propose extending joint independent component analysis (jICA) into spatiotemporal modeling of regional cortical thickness and WM bundle volumes leveraging multimodal MRI. We jointly characterize these GM and WM features in a normal aging (n=316) and an age- and sex-matched preclinical AD cohort (n=81) at each of two imaging sessions spaced three years apart, training on the normal aging population in cross validation and interrogating the preclinical AD cohort. We find this joint model identifies reproducible, longitudinal changes in GM and WM between the two imaging sessions and that these changes are associated with preclinical AD and are plausible considering the literature. We compare this joint model to two focused models: (1) GM features at the first session and WM at the second and (2) vice versa. The joint model identifies components that correlate poorly with those from the focused models, suggesting the different models resolve different patterns. We find the strength of association with preclinical AD is improved in the GM to WM model, which supports the hypothesis that medial temporal and frontal thinning precedes volume loss in the uncinate fasciculus and inferior anterior-posterior association fibers. These results suggest that jICA effectively generates spatiotemporal hypotheses about GM and WM in preclinical AD, especially when specific intermodality relationships are considered a priori

Metal Artifact Reduction (MAR) has been one of the most challenging problems in Computed Tomography (CT) imaging since the technology was invented 50 years ago. Metal implants in patient bodies prevent a CT system from accurately measuring patient anatomies because of the beam hardening effect, X-ray beams’ statistical property changes, and metallic objects’ shapes. Although this problem has been researched for many years, there are still limitations in the conventional model-based methods. Recently, many researchers started using Deep Learning (DL) for the CT MAR problem, which should work better than traditional methods because of its powerful representation capability. In this paper, a new DL-based method is proposed. Our technique is based upon one of the convex optimization methods called primal-dual optimization. A Recurrent Neural Network is adopted to implement the primal-dual optimization for CT MAR. Because of the multi-variable optimization, this neural network conducts dual-domain learning. In addition, the proposed method replaces a metal trace, commonly used in CT MAR methods, with a novel sinogram confidence map. This floating-point map works better than a binary metal trace because of its smoother boundary pixels. Results from extensive experiments indicate that our proposed method outperforms conventional model-based methods and DL-based methods in terms of the Root Mean Squared Error (RMSE), Peak Signal-to-Noise Ratio (PSNR), and Structure Similarity (SSIM) as well as in terms of visual appearance.

Image reconstruction for positron emission tomography (PET) is challenging because of the ill-conditioned tomographic problem and low counting statistics. Kernel methods address this challenge by using kernel representation to incorporate image prior information in the forward model of iterative PET image reconstruction. Existing kernel methods construct the kernels commonly using an empirical procedure, which may lead to suboptimal performance. In this paper, we describe the equivalence between the kernel representation and a trainable neural network model. A deep kernel method is proposed with the training process utilizing available image prior to seek the best way to form a set of robust kernels optimally rather than empirically. The results from computer simulations and a real patient dataset demonstrate that the proposed deep kernel method can outperform existing kernel method and neural network method for dynamic PET image reconstruction.

Muscle, bone, and fat segmentation of CT thigh slice is essential for body composition research. Voxel-wise image segmentation enables quantification of tissue properties including area, intensity and texture. Deep learning approaches have had substantial success in medical image segmentation, but they typically require substantial data. Due to high cost of manual annotation, training deep learning models with limited human labelled data is desirable but also a challenging problem. Inspired by transfer learning, we proposed a two-stage deep learning pipeline to address this issue in thigh segmentation. We study 2836 slices from Baltimore Longitudinal Study of Aging (BLSA) and 121 slices from Genetic and Epigenetic Signatures of Translational Aging Laboratory Testing (GESTALT). First, we generated pseudo-labels based on approximate hand-crafted approaches using CT intensity and anatomical morphology. Then, those pseudo labels are fed into deep neural networks to train models from scratch. Finally, the first stage model is loaded as initialization and fine-tuned with a more limited set of expert human labels. We evaluate the performance of this framework on 56 thigh CT scans and obtained average Dice of 0.979,0.969,0.953,0.980 and 0.800 for five tissues: muscle, cortical bone, internal bone, subcutaneous fat and intermuscular fat respectively. We evaluated generalizability by manually reviewing external 3504 BLSA single thighs from 1752 thigh slices. The result is consistent and passed human review with 5 failed thigh images, which demonstrates that the proposed method has strong generalizability.

Certain body composition phenotypes, like sarcopenia, are well established as predictive markers for post-surgery complications and overall survival of lung cancer patients. However, their association with incidental lung cancer risk in the screening population is still unclear. We study the feasibility of body composition analysis using chest low dose computed tomography (LDCT). A two-stage fully automatic pipeline is developed to assess the cross-sectional area of body composition components including subcutaneous adipose tissue (SAT), muscle, visceral adipose tissue (VAT), and bone on T5, T8 and T10 vertebral levels. The pipeline is developed using 61 cases of the VerSe`20 dataset, 40 annotated cases of NLST, and 851 inhouse screening cases. On a test cohort consisting of 30 cases from the inhouse screening cohort (age 55 - 73, 50% female) and 42 cases of NLST (age 55 - 75, 59.5% female), the pipeline achieves a root mean square error (RMSE) of 7.25 mm (95% CI: [6.61, 7.85]) for the vertebral level identification and mean Dice similarity score (DSC) 0.99 ± 0.02, 0.96 ± 0.03, and 0.95 ± 0.04 for SAT, muscle, and VAT, respectively for body composition segmentation. The pipeline is generalized to the CT arm of the NLST dataset (25,205 subjects, 40.8% female, 1,056 lung cancer incidences). Time-to-event analysis for lung cancer incidence indicates inverse association between measured muscle cross-sectional area and incidental lung cancer risks (p < 0.001 female, p < 0.001 male). In conclusion, automatic body composition analysis using routine lung screening LDCT is feasible.

Lesions or organ boundaries visible through medical imaging data are often ambiguous, thus resulting in significant variations in multi-reader delineations, i.e., the source of aleatoric uncertainty. In particular, quantifying the inter-observer variability of manual annotations with Magnetic Resonance (MR) Imaging data plays a crucial role in establishing a reference standard for various diagnosis and treatment tasks. Most segmentation methods, however, simply model a mapping from an image to its single segmentation map and do not take the disagreement of annotators into consideration. In order to account for inter-observer variability, without sacrificing accuracy, we propose a novel variational inference framework to model the distribution of plausible segmentation maps, given a specific MR image, which explicitly represents the multi-reader variability. Specifically, we resort to a latent vector to encode the multi-reader variability and counteract the inherent information loss in the imaging data. Then, we apply a variational autoencoder network and optimize its evidence lower bound (ELBO) to efficiently approximate the distribution of the segmentation map, given an MR image. Experimental results, carried out with the QUBIQ brain growth MRI segmentation datasets with seven annotators, demonstrate the effectiveness of our approach.

Head and neck cancer is the sixth most common form of cancer in the world population. A commonly used treatment is radiotherapy, which requires physicians to first segment organs at risk (OARs) and tumors in computed tomography images, which is a laborious and time-consuming process. Therefore, a lot of research is being done to develop automatic methods for OAR segmentation. In this paper, we present the results of parotid gland segmentation with nnU-Net using data from two public datasets (Head-Neck-Radiomics-HN1 and PDDCA) and one private dataset acquired at the local hospital. To simulate a possible model deployment scenario, the first model was trained only on publicly available datasets and evaluated on the private dataset, and then compared to the second model that was trained on the same data with additional 10 images from the private dataset. We enrich the interpretation of the results with the comparison among different datasets and among delineations generated with a deep learning model against the delineations of a junior and senior expert that are available for our private dataset. Significant differences were observed among model performance on different datasets, but not among different observers. The performance of nnU-Net on the PDDCA dataset is on par with the state-of-the-art results reported in the literature. Also, the method performed very well compared to the inter-observer variability calculated on our private dataset.

Generative adversarial networks (GANs) have been widely investigated for many potential applications in medical imaging. DatasetGAN is a recently proposed framework based on modern GANs that can synthesize high-quality segmented images while requiring only a small set of annotated training images. The synthesized annotated images could be potentially employed for many medical imaging applications, where images with segmentation information are required. However, to the best of our knowledge, there are no published studies focusing on its applications to medical imaging. In this work, preliminary studies were conducted to investigate the utility of DatasetGAN in medical imaging. Three improvements were proposed to the original DatasetGAN framework, considering the unique characteristics of medical images. The synthesized segmented images by DatasetGAN were visually evaluated. The trained DatasetGAN was further analyzed by evaluating the performance of a pre-defined image segmentation technique, which was trained by the use of the synthesized datasets. The effectiveness, concerns, and potential usage of DatasetGAN were discussed.

Training computer-vision related algorithms on medical images for disease diagnosis or image segmentation is difficult due to the lack of training data, labeled samples, and privacy concerns. For this reason, a robust generative method to create synthetic data is highly sought after. However, most three-dimensional image generators require additional image input or are extremely memory intensive. To address these issues we propose adapting video generation techniques for 3- D image generation. Using the temporal GAN (TGAN) architecture, we show we are able to generate realistic head and neck PET images. We also show that by conditioning the generator on tumour masks, we are able to control the geometry and location of the tumour in the generated images. To test the utility of the synthetic images, we train a segmentation model using the synthetic images. Synthetic images conditioned on real tumour masks are automatically segmented, and the corresponding real images are also segmented. We evaluate the segmentations using the Dice score and find the segmentation algorithm performs similarly on both datasets (0.65 synthetic data, 0.70 real data). Various radionomic features are then calculated over the segmented tumour volumes for each data set. A comparison of the real and synthetic feature distributions show that seven of eight feature distributions had statistically insignificant differences (𝑝 < 0.05). Correlation coefficients were also calculated between all radionomic features and it is shown that all of the strong statistical correlations in the real data set are preserved in the synthetic data set.

Cycle reconstruction regularized adversarial training—e.g., CycleGAN, DiscoGAN, and DualGAN—has been widely used for image style transfer with unpaired training data. Several recent works, however, have shown that local distortions are frequent, and structural consistency cannot be guaranteed. Targeting this issue, prior works usually relied on additional segmentation or consistent feature extraction steps that are task-specific. To counter this, this work aims to learn a general add-on structural feature extractor, by explicitly enforcing the structural alignment between an input and its synthesized image. Specifically, we propose a novel input-output image patches self-training scheme to achieve a disentanglement of underlying anatomical structures and imaging modalities. The translator and structure encoder are updated, following an alternating training protocol. In addition, the information w.r.t. imaging modality can be eliminated with an asymmetric adversarial game. We train, validate, and test our network on 1,768, 416, and 1,560 unpaired subject-independent slices of tagged and cine magnetic resonance imaging from a total of twenty healthy subjects, respectively, demonstrating superior performance over competing methods.

Detection and quantification of atherosclerotic plaque in the coronary arteries is important in cardiovascular risk analysis. Atherosclerotic plaque can be visualized using coronary computed tomography angiography (CCTA). Manual identification and segmentation of plaque is a complex task that requires high level of expertise. Hence, several automatic approaches have been designed. Automatic methods employing deep learning have shown to outperform conventional approaches, but they are hampered by the requirement on the availability of large and diverse training data. To address this, we designed a method that synthesizes calcified and non-calcified atherosclerotic plaque in the coronary arteries. First, we generate plaque geometry using conventional image analysis approach that varies the radius, length and angle of a plaque and we use this to generate a crude inpainting of a plaque on a target artery. Thereafter, we employ a conditional generative adversarial network (GAN) to synthesize the plaque texture in CCTA. The generator is trained to generate fake images with realistic appearance. The discriminator is trained to distinguish the synthesized fake and the real images. The data set for training and evaluation of the plaque synthesis contained CCTA scans of 102 patients (50 training, 52 testing) with manually annotated calcified and non-calcified plaque. To evaluate performance of the synthesis method, we compared CCTA patches with real and synthesized plaque. The evaluation resulted in mean (standard deviation) structural similarity index of 0.99 (0.01), peak signal noise ratio of 73.99 (5.52) and mean absolute error of 5.56 (3.23) HU. To evaluate whether synthesized data enables plaque segmentation, an additional set of CCTA scans of 92 patients without visible plaque was collected. In these scans, plaque was synthesized using the developed approach, containing in total 615 calcified and 544 non-calcified plaque lesions. The synthesized data was used to train a 3D UNet for segmenting calcified and non-calcified plaque lesions. Automatic segmentation which was trained with real data only resulted in Dice coefficients of 0.68 and 0.35 for calcified and non-calcified plaque, respectively. This was significantly improved by pretraining the network with synthetic data and refining it with real data, which resulted in a Dice coefficients of 0.70 (p=0.03) and 0.36 (p=0.02) for calcified and non-calcified plaque, respectively. The results demonstrate that training with CCTA scans with automatically synthesized calcified and non-calcified plaque improves the performance of plaque segmentation.

Due to potential patient anatomy variation during the treatment, proton therapy requires image-guidance systems to minimize treatment uncertainty and ensure conformal doses are precisely delivered to tumor target. This study proposes a feature matching network to generate volumetric images through two orthogonal 2D X-ray images to detect target location during the treatment. The method leverages the kV image system of proton treatment machines to simultaneously acquire two orthogonal 2D images to consistently predict 3D real-time patient anatomy without biasing by motion. The proposed feature mapping network includes four components: a n° kV feature extractor, a n°+90° kV feature extractor, a feature re-alignment, and a depth estimation. The two feature extractors are constructed by two independent convolutional neural networks (CNN) to capture the details of 2D image structures. The re-aliment component ensures that the two orthogonal 2D images are aligned in the same coordinate system. The last part is built by a fully convolutional network (FCN) to inversely transform 2D information to volumetric images. 10 patient cases with 4D CT (100 3D CT images) were used to demonstrate the proposed method. Each 4DCT includes 10 respiratory phases, and the 2D orthogonal images are generated by forward projecting the 3D CT from each phase to planes. Leave-phase-out experiments were performed to ensure that each phase amount of the 10 phases will be used for testing once. Among all 10 patients, the mean absolute error, peak signal-to-noise ratio, and structural similarity index are 83.29 ± 2.83 HU, 16.86 ± 0.16 dB, and 0.83 ± 0.01. The results indicate that the feature matching network can successfully reconstruct the 3D tumor location using 2D projections, which can support treatment decision-making. The proposed method potentially can benefit proton FLASH therapy by offering real-time target localization and patient dose verification to ensure treatment quality.

This paper proposes a super-resolution (SR) method, for performing SR of medical images training on a newly-built lung clinical CT / micro CT dataset. Conventional SR methods are always trained on bicubic downsampled images (LR) / original images (HR) image pairs. However, registration precision between LR and HR images is not satisfying for SR. Low precision of registration results in conventional SR methods’ unsatisfactory performance in medical imaging. We propose a coarse-to-fine cascade framework for performing SR of medical images. First, we design a coarse SR network to translate LR medical images into coarse SR images. Next, we utilize a fully convolutional network (FCN) to perform fine SR (translate coarse SR images to fine SR images). We conducted experiments using a newly-built clinical / micro CT lung specimen dataset. Experimental results illustrated that our method obtained PSNR of 27.30 and SSIM of 0.75, outperforming conventional method’s PSNR 19.08 and SSIM 0.63.

Changes in brain structure and connectivity in aging can be probed through diffusion weighted MRI and summarized with structural connectome matrices. Complex network analysis based on graph theory has been applied to provide measures that are correlated with neurobiological variations and can help guide quantitative study of brain function. However, the understanding of how connectomes change longitudinally is limited. In this work, we evaluate modern pipelines to obtain the connectomics data from diffusion weighted MRI scans across different sessions from control subjects (55-99 years old) in the Baltimore Longitudinal Study of Aging and Cambridge Centre for Ageing and Neuroscience. From the connectivity matrices, we compute graph theory measures to understand their brain networks and apply a linear mixed effects model to study their longitudinal changes with respect to age. With this approach, we computed 14 graph theory measures of 1469 healthy subjects (2476 scans) and found statistically significant correlations between all 14 measures and age. In this analysis: 1) the brain becomes more segregated but less integrated in aging; 2) the overall network cost increases for older subjects; 3) the small-world organizations remain stable; and 4) due to high intra-subject variance, there is not clear trend for longitudinal changes of graph theory measures of individual subjects. Therefore, while useful to investigate brain evolution in aging at the population level, improvements in the connectome reconstruction are needed to decrease single subject variability for individual inference.

Magnetic Resonance Imaging (MRI) encompasses a set of powerful imaging techniques for understanding brain structure and diagnosing pathology. Various MRI sequences including T1- and T2-weighted provide rich complementary information. However, significant equipment costs and acquisition times have inhibited uptake of this critical technology, adversely impacting health equity globally. To ameliorate these costs associated with brain MRIs, we present pTransGAN, a generative adversarial network (GAN) capable of translating both healthy and unhealthy T1 scans into T2 scans, potentially obviating T2 acquisition. Extending prior GAN-based image translation, we show that the addition of non-adversarial losses, like style and content loss, improves the translations provided, especially making the generated images sharper, and making the model more robust. Additionally in previous studies, separate models have been created for healthy and unhealthy brain MRI. Thus here, we also present a novel simultaneous training protocol that allows pTransGAN to concurrently train on healthy and unhealthy data sampled from two open brain MRI datasets. As measured by novel metrics that closely match perceptual similarity of human observers, our simultaneously trained pTransGAN model outperforms the models individually trained on just healthy or unhealthy data. These encouraging results should be further validated with independent paired and unpaired clinical datasets.

In this work, we address a contemporary research problem in the domain of perceptual brain decoding, involving image synthesis from EEG signals in an adversarial deep learning framework. The specific task involves reconstructing images of different object classes, using the EEG recordings acquired when subjects are shown the images of those objects. For this work, we use an EEG encoder for generating EEG encodings. These EEG encodings act as an input to the generator of the GAN network. In addition to the adversarial loss, we also use perceptual loss for generating decent quality images. Through experiments, we demonstrate that the proposed network is generating better quality images than the available state-of-the-art methods.

Given the recent interest in the role of deep generative models (DGM) in medical imaging pipelines, it is imperative to evaluate the capacity of such models to generate medically accurate images. Popular methods of evaluation of natural images generated using generative adversarial networks (GANs), a type of DGM, are often applied to medical data. Such methods are insufficient to evaluate anatomical realism, representations of which include high-order spatial information. To our knowledge, no test exists for the faithful replication of spatial statistics beyond the second-order. In this work, purposefully designed stochastic object models (SOMs) are proposed to encode predetermined rules governing the prevalence of features within single images, thus encoding known high-order spatial information within each realization. These SOMs are independent of the network architecture being tested and can also be applied to any new architecture that may be proposed. Two popular GANs are trained on these SOM datasets and the generated images are tested for the encoded statistics. It is observed that although ensemble statistics might be well replicated, this is not necessarily true for realization i.e., per-image statistics. Thus, GAN-generated images might not be ready for clinical use. With the proposed SOMs, the rate of image errors and the rate of feature malformation can be quantified for any architecture, while providing one measure of GAN utility in a diagnostic scenario.

In medical imaging, obtaining large amounts of labeled data is often a hurdle, because annotations and pathologies are scarce. Anomaly detection is a method that is capable of detecting unseen abnormal data while only being trained on normal (unannotated) data. Several algorithms based on generative adversarial networks (GANs) exist to perform this task, yet certain limitations are in place because of the instability of GANs. This paper proposes a new method by combining an existing method, GANomaly, with progressively growing GANs. The latter is known to be more stable, considering its ability to generate high resolution images. The method is tested using Fashion MNIST, Medical Out-of-Distribution Analysis Challenge (MOOD), and in-house brain MRI; using patches of size 16x16 and 32x32. Progressive GANomaly outperforms a one-class SVM or regular GANomaly on Fashion MNIST. Artificial anomalies are created in MOOD images with varying intensities and diameters. Progressive GANomaly detected the most anomalies with varying intensity and size. Additionally, the intermittent reconstructions are proven to be better from progressive GANomaly. On the in-house brain MRI dataset, regular GANomaly outperformed the other methods.

Aim: To study the impact of the CT scan protocol for Functional Respiratory Imaging (as developed by Fluidda, Belgium) and, where possible, suggest a scan protocol that provides consistent results over different scanners. Methods: In a first step, 76 CT scans have been realized of Kyoto Kagaku’s Lungman, all with different exposure and/or reconstruction settings on two CT scanners. In a second step, all scans were processed to collect pulmonary blood vessel volume using the blood segmentation algorithm of the software platform of Fluidda (Kontich, Belgium). The relative fraction of the blood vessel volume from vessels smaller than 5 mm2 in cross-sectional area (BV5) were compared among the scans, and the variability was calculated. Results: In general, tube current and tube voltage have only a very small influence on BV5. The reconstruction method on the other hand affects the results significantly. The differences between minimal and maximal value per acquisition procedure reach up to 11 % for the proportional BV5 values. The results for the reconstruction methods that make the images smoother than suggested deviate from the other reconstructions. Conclusion: The impact of scan parameters on quantitative data should be verified with a sufficiently dense sample of scan protocols to ensure consistent data. Anthropomorphic phantoms should be developed with gold standard parameters (such as total blood vessel volume) and include all structures of interest. When smooth reconstructions are avoided, consistent data between scanners and acquisition methods can be obtained for the Lungman phantom.

Dual-energy computed tomography (DECT) is a promising technology that has shown a number of clinical advantages over conventional X-ray CT, such as improved material identification, artifact suppression, etc. However, the material decomposition of DECT is an ill-posed inverse problem, which is sensitive to noise. To address this issue, we propose an iterative material decomposition method with gradient L0-norm minimization. The proposed method is accelerated by parallel computing technique. The performance of the proposed method is demonstrated via both phantom and patient studies.

Radiomics is a promising mathematical tool for characterizing disease and predicting clinical outcomes from radiological images such as CT. Photon-counting-detector (PCD) CT provides improved spatial resolution and dose efficiency relative to conventional energy-integrating-detector CT systems. Since improved spatial resolution enables visualization of smaller structures and more details that are not typically visible at routine resolution, it has a direct impact on textural features in CT images. Therefore, it is of clinical interest to quantify the impact of the improved spatial resolution on calculated radiomic features and, consequently, on sample classification. In this work, organic samples (zucchini, onions, and oranges) were scanned on both clinical PCD-CT and EID-CT systems at two dose levels. High-resolution PCD-CT and routine-resolution EID-CT images were reconstructed using a dedicated sharp kernel and a routine kernel, respectively. The noise in each image was quantified. Fourteen radiomic features of relevance were calculated in each image for each sample and compared between the two scanners. Radiomic features were plotted pairwise to evaluate the resulting cluster separation of the samples by their type between PCD-CT and EID-CT. Thirteen out of 14 studied radiomic features were notably changed by the improved resolution of the PCD-CT system, and the cluster separation was better when assessing features derived from PCD-CT. These results show that features derived from high-resolution PCD-CT, which are subject to higher noise compared to EID-CT, may impact radiomics-based clinical decision making.

A unique deep learning network, Deep-E, is proposed, which utilizes 2D training data to solve a 3D problem. The novelty of this simulation method is to generate a 2D matrix in the axial-elevational plane using an arc-shaped transducer element, instead of generating a 3D matrix using the linear transducer arrays. Deep-E exhibited significant resolution improvement on the in vivo human breast data. In addition, we were able to restore deeper vascular structures and remove the noise artifact. We envision that Deep-E will have a significant impact in linear-array-based photoacoustic imaging studies by providing high-speed and high-resolution image enhancement.

In previous deep learning based super-resolution techniques for CT images, only image domain data is used for training. However, image blurring can occur in image domain method which disrupts accurate diagnosis. In this work, we propose using both sinogram and image domain data to resolve the blurring issue. To predict upsampled sinogram more accurately, we use a convolutional neural network (CNN) as an encoder, which maps an input image to feature map for decoder. For decoder, we use dual multi-layer perceptron (MLP) structure. Our proposed dual-MLP structure consists of modulator and synthesizer MLP. Synthesizer MLP predicts the output pixel value by using coordinate-based information as an input, and modulator MLP helps synthesizer to estimate the output value accurately by using feature map information as an input. This network structure preserves high frequency components better than simple CNN structure. Through our proposed sinogram upsampling network (SUN) at sinogram domain, upsampled sinogram was generated, and image was reconstructed by filtered backprojection. The reconstructed image from upsampled sinogram preserves detailed textures compared to LR image. However, residual artifacts and blur still remain. Therefore, we train CNN using image domain data to reduce residual artifacts and blur. For the dataset, we acquire projection data from Mayo Clinic image using Siddon's algorithm in fan-beam CT geometry applying 4x1 detector binning. The binned sinogram is then used as an input for the SUN. The results show that our proposed hybrid domain method outperforms image domain and sinogram domain method with higher quantitative evaluation results.

Extending dedicated breast CT to dynamic contrast-enhanced breast CT will allow functional data by following contrast distribution over multiple time points. As a result of longer scans, unwanted patient motion becomes more likely and motion correction is necessary to avoid artifacts. A crucial part of the development of a motion compensation method is its validation on data with known ground truth. Thus, anthropomorphic phantoms with realistic motion patterns are needed. We present a method to combine motion vector fields, extracted from dynamic contrast-enhanced breast magnetic resonance imaging (DCE-MRI), with digital breast phantoms. DCE-MRI is an ideal source for these data due to its frequent clinical use and a similar prone patient positioning compared to breast CT. Our algorithm consists of three steps. First, the inter-scan motion vector fields are obtained by registration of consecutive images in the DCE-MRI sequence. In the second stage, the digital breast phantoms are aligned on MRI scans using an affine transformation. Finally, the obtained motion vector fields are transformed and applied to the phantoms after parameterization such that the total motion is distributed smoothly in time. The applied motion is evaluated in reconstructions of the simulated breast CT acquisition by qualitative comparison to clinical cases with intra-scan motion. We show that phantoms with simulated motion exhibit the same artifacts as in clinical data such as smooth transitions at the tissue interfaces and ghosting of fine structures.

Speckle-based phase-contrast imaging offers enhanced sensitivity towards weakly-attenuating materials and a simple and cheap setup, but requires accurate tracking of sample-induced speckle pattern modulations. We implemented a convolution neural network for speckle tracking in x-ray phase contrast imaging. The model was trained on simulated speckle patterns generated from a wave-optics simulation and then compared against conventional algorithms. Our solution showed comparable bias, substantially improved root mean squared error and spatial resolution, and the shortest computational time. Thus, our approach enhances the performance of speckle-based phase-contrast imaging.

Clinical data warehouses provide access to massive amounts of medical images and thus offer unprecedented opportunities for research. However, they also pose important challenges, a major challenge being their heterogeneity. In particular, they contain patients with numerous different diseases. The exploration of some neurological diseases with magnetic resonance imaging (MRI) requires injecting a gadolinium-based contrast agent (for instance to detect tumors or other contrast-enhancing lesions) while other diseases do not require such injection. Image harmonization is a key factor to enable unbiased differential diagnosis in such context. Additionally, classical neuroimaging software tools that extract features used as inputs of classification algorithms are typically applied only to images without gadolinium. The objective of this work is to homogenize images from a clinical data warehouse and enable the extraction of consistent features from brain MR images, no matter the initial presence or absence of gadolinium. We propose a deep learning approach based on a 3D U-Net to translate contrast-enhanced into non-contrast-enhanced T1-weighted brain MRI. The approach was trained/validated using 230 image pairs and tested on 26 image pairs of good quality and 51 image pairs of low quality from the data warehouse of the hospitals of the Greater Paris area (Assistance Publique-Hˆopitaux de Paris [AP-HP]). We tested two different 3D U-Net architectures and we chose the one reaching the best image similarity metrics for a further validation for a segmentation task. We tested two 3D U-Net architectures with the addition either of residual connections or of attention mechanisms. The U-Net with attention mechanisms reached the best image similarity metrics and was further validated on a segmentation task. We showed that features extracted from the synthetic images (gray matter, white matter and cerebrospinal fluid volumes) were closer to those obtained from the non-contrast-enhanced T1-weighted brain MRI (considered as reference) than the original, contrast-enhanced, images.

Organ localization is a common and essential preprocessing operation for many medical image analysis tasks. We propose a novel multi-organ localization method based on an end-to-end 3D convolutional neural network. The proposed algorithm employs a regression network to learn the position relationship between any patch and target organs in a medical computed tomography (CT) image. With this framework, it can iteratively localize the target organs in a coarse-to-fine manner. The main idea behind this method is to embed the anatomy of structures in a deep learning-based approach. For implementation, the proposed network outputs an 8-dimensional vector that contains information about the position, scale, and presence of each target organ. A piecewise loss function and a multi-density sampling strategy help to optimize this network to learn anatomy layout characteristics over the entire CT image. Starting from a random position, this network can accurately locate the target organ with a few iterations. Moreover, a dual-resolution strategy is employed to improve the accuracy affected by varying organ scales, further enhancing the localizing performance for all organs. We evaluate our method on a public data set (LiTS) to locate 11 organs in the thoraco-abdomino-pelvic region. The proposed method outperforms state-of-the-art methods with a mean intersection over union (IOU) of 80.84%, mean wall distance of 3.63 mm, and mean centroid distance of 4.93 mm, constituting excellent accuracy. The improvements on relatively small-size and medium-size organs are noteworthy

Image segmentation is the process of delineating the regions occupied by objects of interest in a given image. This operation is a fundamentally required first step in numerous applications of medical imagery. In the medical imaging field, this activity has a rich literature that spans over 45 years. In spite of numerous advances, including deep learning (DL) networks (DLNs) in recent years, the problem has defied a robust, fail-safe, and satisfactory solution, especially for objects that are manifest with low contrast, are spatially sparse, have variable shape among individuals, or are affected by imaging artifacts, pathology, or post-treatment change in the body. Although image processing techniques, notably DLNs, are uncanny in their ability to harness low-level intensity pattern information on objects, they fall short in the high-level task of identifying and localizing an entire object as a gestalt. This dilemma has been a fundamental unmet challenge in medical image segmentation. In this paper, we demonstrate that by synergistically marrying the unmatched strengths of high-level human knowledge (i.e., natural intelligence (NI)) with the capabilities of DL networks (i.e., artificial intelligence (AI)) in garnering intricate details, these challenges can be significantly overcome. Focusing on the object recognition task, we formulate an anatomy-guided DL object recognition approach named Automatic Anatomy Recognition-Deep Learning (AAR-DL) which combines an advanced anatomy-modeling strategy, model-based non-DL object recognition, and DL object detection networks to achieve expert human-like performance.

In recent years, the use of large animals in neurological research has escalated due to advantages over small animals. Unfortunately, large animal imaging researchers lack functional automated medical imaging tools, requiring laborious manual processing. As a response, we have implemented a Reinforcement Learning pipeline for brain anatomical landmark detection in minipig MRIs. Leveraging a deep convolutional network, two-step detection process, and multiple Deep-Q multi-agent networks, our approach is suitable for accurate landmark detection in large animals. Using a heterogeneous dataset containing 154 minipig images, we achieved an average accuracy of 1.56mm on predicting 19 landmarks.

Optical coherence tomography (OCT), a non-invasive high-resolution imaging technology of retinal tissues, has been widely used in the diagnosis of retinal diseases. However, the shortage of ophthalmologists and the overloaded work have caused great difficulties in screening for retinal diseases. Therefore, developing an accurate automatic diagnosis system for screening retinal diseases in OCT images is essential for the prevention and treatment of retinal diseases. To this end, we propose a novel multi-view-based automatic aided diagnosis method for simultaneously screening multiple diseases in retinal OCT images. First, we collected 11,211 cases of 11 common retinal diseases from the ophthalmology clinic, and each case included two OCTs acquired from different views. Then, to automatically and accurately screen diseases in retinal OCT images, a novel multi-view attention network is proposed for screening retinal diseases based on the collected data. Finally, we conduct experiments based on the collected clinical data to evaluate the performance of the proposed method. The AUC of the proposed method achieves 0.9023, which indicates the effectiveness of the proposed method.

During childhood, neurological involvement in tuberous sclerosis complex (TSC) is a leading cause of death. Neurological involvement, including epilepsy, can cause significant long-term sequelae in children. Brain involvement in TSC can be detected by magnetic resonance imaging (MRI). Still, neuroimaging analysis is time-and labor-intensive, begging the need for automated approaches to these tasks to improve speed, accuracy, and availability. We explored the general feasibility of using three-dimensional convolutional neural networks (CNNs) to automatically enhance image diagnosis quality and consistency to identify anatomical abnormalities in TSC children. We trained the 3D CNN on axial T1-weighted, axial T2-weighted FLAIR, and 3DT1-FSPGR weighted images from 296 TSC and 245 Normal cases from birth to 8 years of age acquired at LeBonheur Children’s Hospital. In the best performing approach, we achieved an accuracy of 0.86 [95% CI:0.76-0.97] with 0.95% AUC. The code can be found in https://github.com/shabanian2018/TSC3DCNN

Optical coherence tomography (OCT) is widely used in the diagnosis of retinal diseases. Reading OCT images and summarizing its insights is a routine, yet nonetheless time-consuming task. Automatic report generation can alleviate this issue. There are two major challenges in this task: (1) An OCT image may contain several fundus abnormalities and it is difficult to detect them all simultaneously. (2) The diagnostic reports are complex, which need to describe multiple lesions. In this paper, we propose a deep learning-based model, named as VSTA model (Visual and Semantic Topic Attention model), which is able to generate report from the input OCT image. Our major contributions include: (1) Semantic attention and visual attention are jointly embedded to the model to generate diagnosis report with complex content. (2) Semantic tags based on image similarity is employed to initialize the semantic attention weights, which increases the prediction accuracy of the model. With the proposed VSTA model, the metric of BLEU-4, CIDEr and ROUGE-L reach 31.16, 264.22 and 52.58, which are better than some existing advanced methods.

Determining if the brain is developing normally is a key component of pediatric neuroradiology and neurology. Brain magnetic resonance imaging (MRI) of infants demonstrates a specific pattern of development beyond simply myelination. While radiologists have used myelination patterns, brain morphology and size characteristics to determine age-adequate brain maturity, this requires years of experience in pediatric neuroradiology. With no standardized criteria, visual estimation of the structural maturity of the brain from MRI before three years of age remains dominated by inter-observer and intra-observer variability. A more objective estimation of brain developmental age could help physicians identify many neurodevelopmental conditions and diseases earlier and more reliably. Such data, however, is naturally hard to obtain, and the observer ground truth not much of a gold standard due to subjectivity of assessment. In this light, we explore the general feasibility to tackle this task, and the utility of different approaches, including two- and three-dimensional convolutional neural networks (CNN) that were trained on a fusion of T1-weighted, T2-weighted, and proton density (PD) weighted sequences from 84 individual subjects divided into four age groups from birth to 3 years of age. In the best performing approach, we achieved an accuracy of 0.90 [95% CI:0.86-0.94] using a 2D CNN on a central axial thick slab. We discuss the comparison to 3D networks and show how the performance compares to the use of only one sequence (T1w). In conclusion, despite the theoretical superiority of 3D CNN approaches, in limited-data situations, such approaches are inferior to simpler architectures. The code can be found in https://github.com/shabanian2018/Age_MRI-Classification

Individuals vary in behavior and cognition within a diseased or even “normal” population. Subtyping is critical for characterizing and understanding individual variations in behavior. Neuroimaging-based subtyping has emerged recently and shown great potential for clustering individuals with distinct neural patterns across sub-clusters. However, due to the high dimensional nature of neuroimaging data and relatively limited sample size, commonly used clustering methods in most existing studies such as k-mean may undermine the subtyping results due to the lack of power. Subspace clustering method with elastic net regularization is superior in the sense of jointly learning the sparse affinity matrix and its clustering to preserve reliable high dimension information during clustering limited number of samples. The current study aimed to introduce the Elastic Net subspace clustering to subtype brain structural connectivity matrices robustly. We have included 105 healthy young subjects and constructed structural connectivity matrices based on diffusion tensor imaging as input features for clustering. By calculating and optimizing two indices, “Silhouette Coefficient” and “Calinski-Harabasz” index, optimal parameters were selected to balance the low rank sparse subspace clustering and least square regression and determine the optimal number of clusters. Then the stability of clustering results was tested by subsampling all subjects and clustering each subgroup 3000 times, and the across- and within-subject clustering error rates were estimated. After receiving two robust clusters with high stability, we further explored and found different neural connectivity patterns between clusters. Results suggest that neurosubtyping has the potential to reveal underlying distinct neural patterns speaking to the variations in behavior and neurobehavioral relationships.

For many medical questions, X-ray imaging belongs to the gold standard for diagnosis, treatment planning, treatment guidance, and surgery assessment. To improve the reading performance, standardized image rotation is an important step. We propose a new algorithm to estimate the correct image rotation. For many body regions, one line can be defined that is aligned with the upright orientation of the X-ray image. This line can be, for example, the shaft axis of a long bone or the axis of the spine. In this paper, we propose a strategy to automatically align X-ray images with their standard orientation. In a first step, the heatmap of this line is determined using the segmentation network D-LinkNet. The rotation direction, up to a top-down flip, is obtained by computing the orientation of the main axis of this heatmap. For the orientation computation, we compare PCA and Hu moments. While the PCA requires to threshold the heatmap, Hu moments can be used directly on the output values of the network, preserving the (un)certainty of the segmentation. We compare these two methods with a ResNet-18 for the direct estimation of the image rotation on 220 X-ray images from the MURA dataset showing the wrist in the AP view. With the heatmap-based approach followed by Hu moments analysis, the median absolute error for the angle estimation can be reduced to 0.7° compared to 1.7° by a direct estimation method. PCA suffers from noisy heatmaps for images of bad quality degrading the overall performance of this approach.

Purpose: The choice of input normalization has effects on the generalization and performance of deep neural networks. While this topic is explored for 2D imaging applications, the influence of different normalization techniques on medical imaging modalities, e.g. cone-beam CT (CBCT), differs due to a different value range and distribution. In this paper a good normalization technique for intra-operatively acquired surgical CBCT volumes is presented. Methods: A set of normalization strategies, namely histogram equalization, min-max scaling, z-score normalization, linear look up table (LUT) with clipping and sigmoid function with clipping is compared on a CBCT volume classification task. Results: The results show that a combination of parameterized LUTs and clipping with the range [-710, 1640] HU independent of the underlying intensity histogram provides the best performance for the task at hand. Conclusions: The clipping based normalization technique helps to compress the feature space to the relevant range. By this approach, most of the information about the intensity values of soft tissue and bone is retained. The clipping range presented in this paper is valid for surgical CBCTs.

The presence and thickness of cardiac adipose tissue (CAT) has been extensively correlated with the progression of coronary artery disease. Although CAT is difficult to identify visually with ultrasound relative to other imaging modalities, it is desirable to bolster the efficacy of this modality due to its inexpensiveness and quick acquisition times. This study explores the viability of leveraging the increased visibility of CAT in MR images of the heart to label regions of CAT in ultrasound. This process entails using B-spline interpolation to address the anisotropy of MR volume segmentations, creating a continuous three-dimensional (3D) representation of the patient’s heart muscle and the surrounding CAT. The left myocardium, which is visible in both MRI and ultrasound, is then used to register an ultrasound scan from the same patient with the MRI-derived cardiac model. The intersection of the two-dimensional (2D) ultrasound plane with the 3D MRI-derived model allows regions in the ultrasound to be labelled according to tissue type. Spectral parameters are extracted from ultrasound data from these labelled regions and used to train a tissue type classifier. Using a random forest classifier, this process achieved a tissue classification accuracy of 76.67% for CAT, 80% for myocardium, and 63.33% for blood. The classifier achieved a sensitivity of 60.87% for CAT, 53.85% for myocardium, and 62.5% for blood. The specificity was 86.49% for CAT, 87.23% for myocardium, and 63.88% for blood.

Multi-level multi-modality fusion radiomics is a promising technique with potential to improve the prognostication of cancer. We aim to use advanced fusion techniques on PET and CT images coupled with deep learning (DL) to improve outcome prediction in head and neck squamous cell carcinoma (HNSCC). In our study, 408 HNSCC patients were included from The Cancer Imaging Archive (TCIA) in a multi-center setting. Prognostic outcomes (binary classification) included overall survival (OS), distant metastasis (DM), locoregional recurrence (LR), and progression free survival (PFS). We utilized a DL algorithm with a 17-layer 3D convolutional neural network (CNN) architecture. Prior to training, each image underwent min-max-normalization, image-augmentation by using random rotations (0-20°) to improve the performance and generalizability of our model and followed by 5-fold-cross-validation. We employed 12 datasets, including CT, PET, and 10 image-level fused datasets. The best OS performance was achieved via Discrete-wavelet-transform (DWT) resulting in mean accuracy of 0.93±0.06. The best DM score was achieved via ratio of low-pass pyramid (RLPP), resulting in an accuracy of 0.95±0.02. Optimal LR and PFS scores were achieved using DWT and RLPP for LR, and Laplacian pyramid for PFS, resulting in accuracies of 0.90-0.92. Comparatively, when using a machine learning framework instead of deep learning, we obtained scores of 0.83, 0.90, and 0.87 for the prediction of OS, DM, and LR. Our study demonstrates that our multi-modality fusion techniques performed better than using standalone PET or CT in prognostication of HNSCC patients and that a high level of accuracy can be achieved for prognostication of HNSCC patients when combining multi modality fusion techniques with DL.

We propose a new U-Net-based method for mitosis detection and a semi-automatic image processing algorithm to generate datasets from the H&E- and pHH3- stained tissue images. Instead of manual annotation, which requires not only specialized knowledge but also a lot of labor and time, our dataset generation algorithm is capable of generating precisely labeled datasets that can be easily used as a data expansion for training various kinds of models. Moreover, the proposed U-Net-based mitosis detection model, called GaussUNet, can learn the features of mitotic figures from the images by using novel two-dimensional-Gaussian-distribution-based labels created from the centroid coordinates given by annotations. In addition, we tried to improve the performance of the model by adding false positives obtained from the trained model as the mitosis look-alikes (MLAs) class to the training data. In the experiments, we confirmed the high performance of the proposed method with a simple and efficient model compared to conventional methods.

Microglial cell proliferation in neural tissue (neuroinflammation) occurs during infections, neurological disease, neurotoxicity, and other conditions. In basic science and clinical studies, quantification of microglial proliferation requires extensive manual counting (cell clicking) by trained experts (∼ 2 hours per case). Previous efforts to automate this process have focused on stereology-based estimation of global cell number using deep learning (DL)- based segmentation of immunostained microglial cells at high magnification. To further improve on throughput efficiency, we propose a novel approach using snapshot ensembles of convolutional neural networks (CNN) with training using local images, i.e., low (20x) magnification, to predict high or low microglial proliferation at the global level. An expert uses stereology to quantify the global microglia cell number at high magnification, applies a label of high or low proliferation at the animal (mouse) level, then assigns this global label to each 20x image as ground truth for training a CNN to predict global proliferation. To test accuracy, cross validation with six mouse brains from each class for training and one each for testing was done. The ensemble predictions were averaged, and the test brain was assigned a label based on the predicted class of the majority of images from that brain. The ensemble accurately classified proliferation in 11 of 14 brains (∼ 80%) in less than a minute per case, without cell-level segmentation or manual stereology at high magnification. This approach shows, for the first time, that training a DL model with local images can efficiently predict microglial cell proliferation at the global level. The dataset used in this work is publicly available at: tinyurl.com/20xData-USF-SRC.

We describe a method for analyzing geometrical properties of cell nuclei from phase contrast microscopy images. This is useful in drug discovery for quantifying the effect of candidate chemical compounds, bypassing the need for fluorescence imaging. Fluorescence images are then only used for training our nuclei segmentation, avoiding the need for the time consuming expert annotations. Geometry based descriptors are calculated and aggregated and fed into a classifier to distinguish the different types of chemical treatments. The drug treatment can be distinguished from no treatment with accuracy better than 95% from fluorescence images and better than 77% from phase contrast images.

Contrastive learning, a recent family of self-supervised learning, leverages pathological image analysis by learning from large-scale unannotated data. However, the state-of-the-art contrastive learning methods (e.g., SimCLR, BYOL) are typically limited by the more expensive computational hardware (with large GPU memory) as compared with traditional supervised learning approaches in achieving large training batch size. Fortunately, recent advances in the machine learning community provide multiple approaches to reduce GPU memory usage, such as (1) activation compressed training, (2) In-place activation, and (3) mixed precision training. Yet, such approaches are currently deployed independently without systematical assessments for contrastive learning. In this work, we applied these memory-efficient approaches into a self-supervised framework. The contribution of this paper is three-fold: (1) We combined previously independent GPU memory-efficient methods with self-supervised learning framework; (2) Our experiments are to maximize the memory efficiency via limited computational resources (a single GPU); (3) The self-supervised learning framework with GPU memory-efficient method allows a single GPU to triple the batch size that typically requires three GPUs. From the experimental results, contrastive learning model with larger batch size leads to higher accuracy enabled by GPU memory-efficient method on single GPU.

Serial-section electron microscopy is a widely used technique for neuronal circuit reconstruction. However, the continuity of neuronal structure is destroyed when the tissue block is cut into a series of sections. The neuronal morphology in different sections changes with their locations in the tissue block. These content changes in adjacent sections bring a difficulty to the registration of serial electron microscopy images. As a result, the accuracy of image registration is strongly influenced by neuronal structure variation and section thickness. To evaluate registration performance, we use the spherical deformation model as a simulation of the neuron structure to analyze how registration accuracy is affected by section thickness and neuronal structure size. We mathematically describe the trend that the correlation of neuronal structures in two adjacent sections decreases with section thickness. Furthermore, we demonstrate that registration accuracy is negatively correlated with neuronal structure size and section thickness by analyzing the second-order moment of estimated translation. The experimental results of registration on synthetic data demonstrate that registration accuracy decreases with the neuronal structure size.

Gleason Score (GS) is the principal histological grading system to support the quantification of cancer aggressiveness. This analysis is carried out by expert pathologists but with reported evidence of moderate agreement among pathologists (kappa values less than 0.5). This fact can be prone to errors that directly affect the diagnosis and subsequent treatment. Current deep learning approaches have been proposed to support such visual pattern quantification but there exist a remarked on expert annotations that overfit representations. Besides, the supervised representation is limited to model the high reported visual variability intra Gleason grades. This work introduces a semi-Supervised Learning (SSL) approach that initially uses a reduced set of annotated visual patterns to built several GS deep representations. Then, the set of deep models automatically propagates annotations to unlabeled patches. The most confident predicted samples are used to retrain the ensemble deep representation. Over a patch-based framework with a total of 26259 samples, coded from 886 tissue microarrays, the proposed approach achieved remarkable results between grades three and four. Interestingly, the proposed SSL with only the 10% of samples achieves more general representation, achieving averages scores of ~75.93% and ∼ 71.88% concerning two expert pathologists

Understanding the brain connectome and the anatomical organization of neural circuits in the mouse brain using histological sections is a prominent area of research in the neuroscience field. Accurate quantitative and comparative analysis of anatomical data requires precise mapping of brain sections to a common reference atlas. The existing methods rely either on using 2D coronal atlases or 3D reconstruction prior to registration. The problem with the former is that atlases are not always a good match, since they do not account for the slicing angle. The drawback of the latter is that 3D to 3D registration methods are not only computationally expensive but also require a full set of consecutive sections which are not always available due to technical limitations. In this study, we propose a deep learning-based approach, to automatically detect the position and angle of individual mouse brain sections in the 3D reference atlas. The novel method is implemented as a pipeline consisting of 3 blocks of Convolutional Neural Network (CNN) regression models that detect the slicing angle and the position of the section in the anterior-posterior (AP) axis of the brain. The proposed method not only generates matching 2D atlases by taking the slicing angle into account but is also considerably faster and more robust to histological artifacts, compared to 3D registration approaches. We have shown that predictions of our method are comparable to a neuroscientist expert.

The Corpus callosum (CC) is a massive white matter structure in the brain, and changes in its shape and volume are associated with subject characteristics, several diseases, and clinical conditions. The CC is mostly studied in magnetic resonance imaging (MRI), where it is segmented to extract valuable information. With the increasing availability of MRI data and the proliferation of automated algorithms to perform CC segmentation, quality control (QC) verification is mandatory to assure reliability in the entire analysis pipeline. We propose a convolutional neural network (CNN) for QC of CC segmentations. The CNN gets information on the mask and contextual information on the image and performs deep feature extraction using a pre-trained model. The CNN model was fine-tuned using T1-weighted MR images with CC masks, in the task of classifying correct or incorrect segmentations. The CNN-based approach got an area under the curve (AUC) of 97.98% on the test set. We used an additional test set of patients with tumor to test generalization capability of the trained model to other domains.

Non-linear registration models optimize two conflicting objectives, a content-matching term and a deformation smoothness measure. As the desired smoothness regime is problem-specific, there is a need to better compare generic registration algorithms across different smoothness regimes. We propose to compare registration algorithms by estimating their content-matching vs deformation smoothness Pareto front. Specifically, we assess the deformation smoothness level reached by each algorithm at different content-matching levels. We introduce a new objective function to sample the Pareto front along a specific iso-content-matching line. We demonstrate the applicability of our method on chest-CT inter-patient registration by comparing 5 learning-based registration algorithms.

The Human BioMolecular Atlas Program (HuBMAP) provides an opportunity to contextualize findings across cellular to organ systems levels. Constructing an atlas target is the primary endpoint for generalizing anatomical information across scales and populations. An initial target of HuBMAP is the kidney organ and arterial phase contrast-enhanced computed tomography (CT) provides distinctive appearance and anatomical context on the internal substructure of kidney organs such as renal context, medulla, and pelvicalyceal system. With the confounding effects of demographics and morphological characteristics of the kidney across large-scale imaging surveys, substantial variation is demonstrated with the internal substructure morphometry and the intensity contrast due to the variance of imaging protocols. Such variability increases the level of difficulty to localize the anatomical features of the kidney substructure in a well-defined spatial reference for clinical analysis. In order to stabilize the localization of kidney substructures in the context of this variability, we propose a high-resolution CT kidney substructure atlas template. Briefly, we introduce a deep learning preprocessing technique to extract the volumetric interest of the abdominal regions and further perform a deep supervised registration pipeline to stably adapt the anatomical context of the kidney internal substructure. To generate and evaluate the atlas template, arterial phase CT scans of 500 control subjects are de-identified and registered to the atlas template with a complete end-to-end pipeline. With stable registration to the abdominal wall and kidney organs, the internal substructure of both left and right kidneys are substantially localized in the high-resolution atlas space. The atlas average template successfully demonstrated the contextual details of the internal structure and was applicable to generalize the morphological variation of internal substructure across patients.

Reliably and physically accurately transferring information between images through deformable image registration with large anatomical differences is an open challenge in medical image analysis. Most existing methods have two key shortcomings: first, they require extensive up-front parameter tuning to each specific registration problem, and second, they have difficulty capturing large deformations and content mismatches between images. There have however been developments that have laid the foundation for potential solutions to both shortcomings. Towards the first shortcoming, a multi-objective optimization approach using the Real-Valued Gene-pool Optimal Mixing Evolutionary Algorithm (RV-GOMEA) has been shown to be capable of producing a diverse set of registrations for 2D images in one run of the algorithm, representing different trade-offs between conflicting objectives in the registration problem. This allows the user to select a registration afterwards and removes the need for up-front tuning. Towards the second shortcoming, a dual-dynamic grid transformation model has proven effective at capturing large differences in 2D images. These two developments have recently been accelerated through GPU parallelization, delivering large speed-ups. Based on this accelerated version, it is now possible to extend the approach to 3D images. Concordantly, this work introduces the first method for multi-objective 3D deformable image registration, using a 3D dual-dynamic grid transformation model based on simplex meshes while still supporting the incorporation of annotated guidance information and multi-resolution schemes. Our proof-of-concept prototype shows promising results on synthetic and clinical 3D registration problems, forming the foundation for a new, insightful method that can include bio-mechanical properties in the registration.

We demonstrate the application of the Thin Shell Demons (TSD) surface registration algorithm in registering the dental scans obtained from intra-oral scanners (IOS) and Cone Beam Computed Tomography (CBCT) in a semi-automatic manner. The reconstructed dentition obtained from CBCT lacks the accuracy for diagnosis and appliance fabrication that IOS provides. Current methods to register IOS to CBCT typically use Iterative Closest Point (ICP) but suffer from a lack of precision and accuracy. TSD registration has previously been shown to produce superior registration results in presence of missing patches and holes. In this work, for the first time we share its application in dental scan registration. We perform experiments on dental surface meshes obtained from CBCT and IOS for two patients who have undergone orthognathic surgery. Our method first registers the IOS mesh with the CBCT mesh using ICP. To obtain tight alignment of the tooth surface, we perform a refinement using the TSD registration. We quantify the improvement in registration by measuring the distance between the closest points present on the surface of six teeth in the IOS and CBCT meshes. Compared to using only ICP registration, TSD decreases the mean surface distance between patches and significantly improves the alignment. We also share qualitative results that clearly demonstrate the improvement due to TSD. For wide adoption and ease of access, we also share a publicly available Thin Shell Demons implementation in C++ under the open source image processing library Insight Toolkit (ITK). Python wrapping of the code is also made available. The code is available at the url: https://github.com/InsightSoftwareConsortium/ITKThinShellDemons.

Deep learning algorithms have become the golden standard for segmentation of medical imaging data. In most works, the variability and heterogeneity of real clinical data is acknowledged to still be a problem. One way to automatically overcome this is to capture and exploit this variation explicitly. Here, we propose an approach that improves on our previous work in this area and explain how it potentially can improve clinical acceptance of (semi-)automatic segmentation methods. In contrast to a standard neural network that produces one segmentation, we propose to use a multi-path Unet network that produces multiple segmentation variants, presumably corresponding to the variations that reside in the dataset. Different paths of the network are trained on disjoint data subsets. Because a priori it may be unclear what variations exist in the data, the subsets should be automatically determined. This is achieved by searching for the best data partitioning with an evolutionary optimization algorithm. Because each network path can become more specialized when trained on a more homogeneous data subset, better segmentation quality can be achieved. In practical usage, various automatically produced segmentations can be presented to a medical expert, from which the preferred segmentation can be selected. In experiments with a real clinical dataset of CT scans with prostate segmentations, our approach provides an improvement of several percentage points in terms of Dice and surface Dice coefficients compared to when all network paths are trained on all training data. Noticeably, the largest improvement occurs in the upper part of the prostate that is known to be most prone to inter-observer segmentation variation.

Dental offices tackle thousands of dental reconstructions every year. Complexity and abnormalities in dentition make segmentation of an optical scan a challenging manual task that takes 45 minutes on average. The present work improves the generalization of currently available deep learning segmentation model on 3D dental arches by introducing a new loss function to leverage unlabeled available data. The semi-supervised segmentation network is trained using a joint loss that combines a supervised loss of annotated input and a self-supervised loss of non-labeled input. Our results showed that combining self-supervised and supervised learning improved the segmentation score by 13 % compared with purely supervised learning for the same amount of labeled data. It is concluded that combining representations obtained from self-supervised learning with supervised learning improves the generalization of the 3D tooth segmentation model in the case of few available labeled data.

Introduction: Vascular diseases, such as intracranial aneurysms, are one of the top causes of death in the world. Due to the constantly increasing number of angiographic imaging examinations and their use in population screening there is a need for accurate and robust methods for vessel segmentation. Methods & Materials: We used a publicly available dataset of 570 cerebral TOF-MRA angiograms (IXI dataset) and manually created reference segmentations using interactive thresholding of the raw and vesselness filter enhanced angiograms. The obtained segmentations were visually verified by a skilled radiologist and then used to objectively and comparatively evaluate six approaches based on recent convolutional neural network (CNN) segmentation models. Results: Model training on raw images (without preprocessing) resulted in Dice similarity coefficient (DSC) value of 0.91, while preprocessing with specialized filters produced inferior DSC values. Spatially affixed model training on the Circle of Willis (CoW) region yielded a significantly better result (DSC=0.95; p-value < 0.001) as compared to the training on whole images (DSC=0.91). Conclusion: On the MRA scans of IXI dataset we created reference vessel segmentations to serve as a new benchmark for vessel segmentation studies. The reference segmentations are publicly available**. Among six state-of-the-art approaches evaluated on this dataset, we found that raw input images with spatially affixed CNN model training with respect to CoW achieved the best vessel segmentation.

Temporal patterns of cardiac motion provide important information for cardiac disease diagnosis. This pattern could be obtained by three-directional CINE multi-slice left ventricular myocardial velocity mapping (3Dir MVM), which is a cardiac MR technique providing magnitude and phase information of the myocardial motion simultaneously. However, long acquisition time limits the usage of this technique by causing breathing artifacts, while shortening the time causes low temporal resolution and may provide an inaccurate assessment of cardiac motion. In this study, we proposed a frame synthesis algorithm to increase the temporal resolution of 3Dir MVM data. Our algorithm is featured by 1) three attention-based encoders which accept magnitude images, phase images, and myocardium segmentation masks respectively as inputs; 2) three decoders that output the interpolated frames and corresponding myocardium segmentation results; and 3) loss functions highlighting myocardium pixels. Our algorithm can not only increase the temporal resolution 3Dir MVMs, but can also generates the myocardium segmentation results at the same time.

Fracture fixation surgeries require a careful and well thought out surgical plan, mainly due to the wide range of possibilities in the fracture types and available choices in fixation constructs. There is considerable interest in virtual 3D planning tools ranging from 3D visualization, interactive fracture reduction and bio-mechanical analysis of fracture fixation construct stability to arrive at optimal plan. One of the key steps prior to reconstructing 3D fractures is accurate fracture segmentation which can be tedious and time consuming even with semi-automated tools. In this paper, we report preliminary results from our attempt to fully automate the segmentation of fractured bone using deep learning. We performed experiments using the widely used 3D segmentation model called 3D U-Net on a dataset of 14 CT volumes. The dataset is randomly divided into train, validation and test splits comprising 7, 3 and 4 volumes respectively. Even with a small training set of femur fractures, we were able to achieve a mean dice score of 0.861 with a mean sensitivity of 0.899. The model was able to capture the challenging fracture regions and could cleanly separate the femur head and socket. Apart from this, we also studied the impact of different loss functions on the network’s performance. The results indicate that deep learning based segmentation methodologies have good potential in automating the challenging task of fractured femur segmentation. Further improvement is expected with a larger collection of such fractured samples

Colorectal cancer (CRC) is one of the most common fatal cancer in the world. Polypectomy can effectively interrupt the progression of adenoma to adenocarcinoma. Colonoscopy is the primary method to find colonic polyps. However, due to the different sizes and the unclear boundary of polyps, it is challenging to segment polyps accurately. To this end, we design a Boundary Distribution Guided Network (BDG-Net) for accurate polyp segmentation. Specifically, Boundary Distribution Generate Module (BDGM) aggregates high-level features to generate Boundary Distribution Map (BDM), which is sent to the Boundary Distribution Guided Decoder (BDGD) as complementary spatial information to guide the polyp segmentation. Moreover, a multi-scale feature interaction strategy is adopted in BDGD to improve the polyp segmentation of different sizes. Extensive experiments demonstrate that BDG-Net outperforms state-of-the-art models remarkably and maintains low computational complexity.

We demonstrate a new approach for blind domain adaptation by employing classic feature descriptors as a first step in a deep learning pipeline. One advantage of our approach over other domain adaptation methods is that no target domain data are required. Therefore, the trained models perform well on a multitude of different datasets as opposed to one specific target dataset. We test our approach on the task of abdominal CT and MR organ segmentation and transfer the models from the training dataset to multiple other CT and MR datasets. We show that modality independent neighborhood descriptors applied prior to a DeepLab segmentation pipeline can yield high accuracies when the model is applied on other datasets including those with a different imaging modality.

This paper evaluates deep learning methods on segmentation of dental arches in panoramic radiographs. Our main aim is to test whether introducing auxiliary learning goals can improve image segmentation. We implement three multi-output networks that detect (1) patient characteristics (e.g missing teeth, no dental artifacts), (2) buccal area, (3) individual teeth, alongside the dental arches. These design choices may restrict the region of interest and improve the internal representation of teeth shapes. The models are based on the modified Unet1 architecture and optimized with Dice loss. Two data sets, of 1500 and 116 samples, collected at different institutions,2, 3 were used for training and testing the methods. Additionally, we evaluated the networks against various patient conditions, namely: 32 teeth, ̸= 32 teeth, dental artifacts, no dental artifacts. The standard U-net architecture reaches the highest Dice scores of 0.932 on the larger data set2 and 0.946 on the group of patients with no missing teeth. The model that outputs probability masks for individual teeth reaches the best Dice score of 0.903 on the smaller data set.3 We observe certain benefits in augmenting teeth segmentation with other information sources, which indicate the potential of this research direction and justifies further investigations.

Organ at risk (OAR) segmentation is a crucial step for treatment planning and outcome determination in radiotherapy treatments of cancer patients. Several deep learning based segmentation algorithms have been developed in recent years, however, U-Net remains the de facto algorithm designed specifically for biomedical image segmentation and has spawned many variants with known weaknesses. In this study, our goal is to present simple architectural changes in U-Net to improve its accuracy and generalization properties. Unlike many other available studies evaluating their algorithms on single center data, we thoroughly evaluate several variations of U-Net as well as our proposed enhanced architecture on multiple data sets for an extensive and reliable study of the OAR segmentation problem. Our enhanced segmentation model includes (a)architectural changes in the loss function, (b)optimization framework, and (c)convolution type. Testing on three publicly available multi-object segmentation data sets, we achieved an average of 80% dice score compared to the baseline U-Net performance of 63%.

Quantification and segmentation of the mid-thigh region has high clinical importance for assessment of muscle composition and adipose tissue depositions. Changes in body composition may characterize chronic diseases like obesity, metabolic disorders, type-2 diabetes, and osteoarthritis. Effective methods for segmentation of soft and hard tissues in the mid-thigh in help to understand and characterize changes caused by disease or normal aging. The purpose of our research is to develop a fully automated system for segmentation of hard and soft tissues from CT scans of the mid-thigh region. In particular, we aim to segment the muscle, intermuscular adipose tissue, and subcutaneous adipose tissue using a deep network. A major challenge in deep learning is to provide a rich and diverse set of data for training. Another limitation in tissue segmentation applications is class imbalance, because larger structures may dominate the training process and introduce classification bias. We propose an adaptive re-sampling method according to the tissue type to address class imbalance. We evaluated the segmentation accuracy of the network by cross-validation techniques using CT scans obtained from the BLSA study. We obtained an overall DSC score of 91.5% for segmentation of the mid-thigh regional tissues. Performance evaluation results leads to the observation that our method produces very good accuracy rates and is competitive with current methods used for quantification. This method applied deep learning to a meaningful clinical application that is not revisited frequently.

The placenta is an important organ for the material exchange between the fetus and the mother. The abnormal placenta may lead to fetal intrauterine growth restriction, invasive placenta and other related diseases, thus endangering the health of both the mother and fetus. Accurate segmentation of placental tissue in fetal magnetic resonance images could help diagnose placental abnormalities. However, the manual segmentation of placenta is very time-consuming, and the semi automatic placental segmentation methods still require the operator’s interaction. In this paper, we proposed a fully automatic placental segmentation method, in which BiO-Net is used as the backbone network and is further improved by embedding Atrous Spatial Pyramid Pooling (ASPP) and attention mechanism, termed as BAA-Net. To retain more details of boundary information, the ASPP module was introduced to the encoder for capturing high-resolution feature maps to improve the placental segmentation performance. Because different feature channels from the encoder and decoder have different effects on the segmentation task, to make better use of the most useful features, four channel attention modules were introduced into the decoder to highlight the most relevant feature channels. To evaluate the performance of the proposed BAA-Net, MR images of 20 pregnant women were used for the experiments. The Dice and average symmetric surface distance (ASSD) obtained by our BAA-Net are 0.8674 and 2.8880 mm, respectively. The experimental results show that the proposed BAA-Net is effective for the automatic placental segmentation, comparing with the existing methods. Accurate placental segmentation is conducive to the diagnosis of placental abnormalities, which has important clinical significance.

Segmentation of digital mammograms (DMs) into background, breast, and pectoral muscle is an important pre-processing step for many medical imaging pipelines. Our aim is to propose a segmentation method suited for processed DMs that generalizes across cranio-caudal (CC) and medio-lateral oblique (MLO) projections, and across models of different vendors. A dataset of 247 diagnostic DM exams was used, totaling 493 CC and 494 MLO processed images, of which 199 (40.4%) and 486 (98.4%) contained a pectoral muscle, respectively. The images were acquired with 10 different DM models from GE (73%) and Siemens (27%). The multi-class segmentation was done by a U-Net trained with a multi-class weighted focal loss. Several types of data augmentation were used during training, to generalize across model types, including random look-up table and random elastic and gamma transformations. The DICE coefficients for the segmentations were (mean ± std. dev.) 0.995 ± 0.005, 0.980 ± 0.016, 0.839 ± 0.243 for background, breast, and pectoral muscle, respectively. Background segmentation did not differ significantly between CC and MLO images. The pectoral muscle segmentation resulted in a higher DICE coefficient for MLO (0.932 ± 0.104) than CC images (0.636 ± 0.323). The false positive rate of pectoral muscle segmentation was 1.5% in CC images without any pectoral muscle. Among different model types, the mean overall DICE coefficients ranged from 0.985-0.990 for the different system models. The developed method yielded accurate overall segmentation results, independent of view, and was able to generalize well over mammograms acquired by systems of different vendors.

Medical image segmentation annotated by experts provides the labeled data sets for many scientific researches. However, due to the unevenly experienced backgrounds of the experts and limited numbers of patients with certain diseases or illnesses, not only do such labeled data sets have smaller samples but their quality and normality also can range in wide variabilities and be ambiguous. In practice, these segmentations are usually assigned to be the ground truths for the scientific studies, so it may undermine the trustworthiness of the resulting findings. Therefore, it is meaningful to consider how to give a more unified opinion of the annotations among different experts. In this paper, a novel approach to form normal distributions of segmentation is proposed based on multiple doctors’ annotations for the same patient. The proposed approach is developed through the following steps: (1) utilize a framework7 of averaging images to construct an averaged annotation based on different given annotations; (2) determine the image registration deformations from the averaged annotation to the given annotations; (3) build a joint multivariate Gaussian distribution over the logorithm of Jacobian determinants and curls of the registration deformations; lastly, (4) simulate a normal distribution of segmentation by the joint Gaussian distribution of registration deformation. This work translates the problem of forming a normal distribution of the image segmentation into a problem of forming joint Gaussian distribution of image registration deformations, which the latter can be reasoned by Jacobian determinant (models local size of pixel cells) and curl (models local rotation of pixel cells) information. In the following sections, a detailed walk-through of the proposed approach is provided along with its analytical mathematics and numerical examples for its effectiveness. A synthetic example of 3 manually defined label image is made to show how to construct a mean label image, and an example of a real cancer image annotated by 3 doctors demonstrates the formation of the normal distribution and the effectiveness of the propose method

dentification of the neck of unruptured intracranial aneurysms is essential for quantitative analysis and monitoring of disease progression. Given a limited number of available tools for neck detection and measurements, physicians have little choices when it comes to producing clinically accurate metrics. This research presents a novel method of aneurysm neck detection based on semi-automated segmentation of aneurysm and its associated vessels. The reported aneurysm segmentation method utilizes LOGISMOS-JEI1 as its foundation to form a novel global-optimality-guaranteeing strategy for quantitative analysis of aneurysm morphology.

In this work, we compare multiple end-to-end neural networks that classify and segment numerous anatomies in fetal torso ultrasound (US) images. The novelty of this paper is not restricted by the fact that it extends the scarce literature on the recently proposed nnUNet approach, we are also the first who apply this framework on 2D US data and compare it with various state-of-the-art 2D segmentation models. Our fetal torso dataset comprises two planes – the four chambers of the heart and the three vessel trachea view – with distinct, however, non-mutually exclusive sets of anatomies, which poses another level of complexity. Consequently, besides segmenting observable anatomies, classifying the absence of such anatomies is of crucial importance for researchers and practitioners as well. We find that the nnUNet outperforms numerous state-of-the-art models both in the classification as well as the segmentation task. In more detail, our findings indicate that the nnUNet achieves the highest scores among all evaluation metrics. Finally, we discuss the benefits of the nnUNet and address potential drawbacks of its design regarding 2D segmentation.

We aimed to develop a novel deep-learning based method for automatic coronary artery calcium (CAC) quantification in low-dose ungated computed tomography attenuation correction maps (CTAC). In this study, we used convolutional long-short -term memory deep neural network (conv-LSTM) to automatically derive coronary artery calcium score (CAC) from both standard CAC scans and low-dose ungated scans (CT-attenuation correction maps). We trained convLSTM to segment CAC using 9543 scans. A U-Net model was trained as a reference method. Both models were validated in the OrCaCs dataset (n=32) and in the held-out cohort (n=507) without prior coronary interventions who had CTAC standard CAC scan acquired contemporarily. Cohen’s kappa coefficients and concordance matrices were used to assess agreement in four CAC score categories (very low: <10, low:10-100; moderate:101-400 and high <400). The median time to derive results on a central processing unit (CPU) was significantly shorter for the conv-LSTM model- 6.18s (inter quartile range [IQR]: 5.99, 6.3) than for UNet (10.1s, IQR: 9.82, 15.9s, p<0.0001). The memory consumption during training was much lower for our model (13.11Gb) in comparison with UNet (22.31 Gb). Conv-LSTM performed comparably to UNet in terms of agreement with expert annotations, but with significantly shorter inference times and lower memory consumption

Image segmentation tasks are considered resource intensive. These tasks require domain specialists to labor manually over long periods of time. When considering medical image segmentation tasks, the personnel and the error margin make these tasks expensive. Therefore there is a need for an automated tool. Deep learning has fast become the state of the art for such tasks, yet the methods applied require large data-sets of fully annotated examples. The need for supervision prevents researchers from developing deep learning and machine learning solutions on new datasets, which were not annotated by professional personnel. In this paper we utilize weak supervision to train a deep neural network to perform instance segmentation. The data used for this project is the Multimodal Brain Tumor Segmentation Challenge 3D MRI scans. The method used is a two-step DNN. The first step is binary classification of slices to either pathological or healthy. This is the only step which uses supervision for the training of the DNNs. In the second step, another DNN in the form of a Unet encoder-decoder network is utilized. This network encodes the input raw data and decodes each pixel to a 32 dimensional vector representing a semantic identity (semantic map). The supervision for training this second network is derived from the GradCAM of the classification DNN. Lastly, to segment the input data we determine the semantic distance between suspected lesion points and the entirety of the map. We achieve an average Dice score of 0.73 over three test sets of 38 patients each.

Accurate segmentation of infant brain magnetic resonance images is crucial for studying brain development. However, infant images even within a narrow age-range differ drastically in size and contrast. Here, we investigated whether deep-learning based methods evaluated in iSeg-2017 can be generalized to accurately segment brain data acquired from other cohorts with different acquisition and ages. ISeg-2017 and UNC infant datasets were used to investigate three methods: SemiDenseNet, HyperDenseNet, and 3D-DenseSeg. Results demonstrate that HyperDenseNet has better segmentation performance and generalizability. Moreover, we built a joint segmentation-registration method by applying HyperDenseNet for segmentation. Results show that the joint method produced better performance compared with only registration or segmentation.

Segmenting full vessel systems of the human liver is important for many applications in liver surgery and intervention planning. While methods exist for training DNNs for vessel segmentation, no method we know of efficiently extracts the vessel graph without modifying the DNN architecture. We demonstrate a fully automatic method for extracting and separating vessel graphs directly from the output of a segmentation model by applying a modified algorithm for vessel connectivity analysis. This method significantly improves the centerline sensitivity of reconstructed graphs on the IRCAD dataset and achieves similar scores for splitting vessel systems as the recently published TopNet approach.

Abdominal computed tomography CT imaging enables assessment of body habitus and organ health. Quantification of these health factors necessitates semantic segmentation of key structures. Deep learning efforts have shown remarkable success in automating segmentation of abdominal CT, but these methods largely rely on 3D volumes. Current approaches are not applicable when single slice imaging is used to minimize radiation dose. For 2D abdominal organ segmentation, lack of 3D context and variety in acquired image levels are major challenges. Deep learning approaches for 2D abdominal organ segmentation benefit by adding more images with manual annotation, but annotation is resource intensive to acquire given the large quantity and the requirement of expertise. Herein, we designed a gradient based active learning annotation framework by meta-parameterizing and optimizing the exemplars to dynamically select the 'hard cases' to achieve better results with fewer annotated slices to reduce the annotation effort. With the Baltimore Longitudinal Study on Aging (BLSA) cohort, we evaluated the performance with starting from 286 subjects and added 50 more subjects iteratively to 586 subjects in total. We compared the amount of data required to add to achieve the same Dice score between using our proposed method and the random selection in terms of Dice. When achieving 0.97 of the maximum Dice, the random selection needed 4.4 times more data compared with our active learning framework. The proposed framework maximizes the efficacy of manual efforts and accelerates learning.

Image segmentation has been increasingly applied in medical settings as recent developments have skyrocketed the potential applications of deep learning. Urology, specifically, is one field of medicine that is primed for the adoption of a real-time image segmentation system with the long-term aim of automating endoscopic stone treatment. In this project, we explored supervised deep learning models to annotate kidney stones in surgical endoscopic video feeds. In this paper, we describe how we built a dataset from the raw videos and how we developed a pipeline to automate as much of the process as possible. For the segmentation task, we adapted and analyzed three baseline deep learning models – U-Net, U-Net++, and DenseNet – to predict annotations on the frames of the endoscopic videos with the highest accuracy above 90%. To show clinical potential for real-time use, we also confirmed that our best trained model can accurately annotate new videos at 30 frames per second. Our results demonstrate that the proposed method justifies continued development and study of image segmentation to annotate ureteroscopic video feeds.

Segmentation of vertebrae and intervertebral discs (IVD) in MR images are important steps for automatic image analysis. This paper proposes an extension of an iterative vertebra segmentation method that relies on a 3D fully-convolutional neural network to segment the vertebrae one-by-one. We augment this approach with an additional segmentation step following each vertebra detection to also segment the IVD below each vertebra. To train and test the algorithm, we collected and annotated T2-weighted sagittal lumbar spine MR scans of 53 patients. The presented approach achieved a mean Dice score of 93 % ± 2 % for vertebra segmentation and 86 % ± 7 % for IVD segmentation. The method was able to cope with pathological abnormalities such as compression fractures, Schmorl’s nodes and collapsed IVDs. In comparison, a similar network trained for IVD segmentation without knowledge of the adjacent vertebra segmentation result did not detect all IVDs (89 %) and also achieved a lower Dice score of 83 % ± 9 %. These results indicate that combining IVD segmentation with vertebra segmentation in lumbar spine MR images can help to improve the detection and segmentation performance compared with separately segmenting these structures.

Deep inspiration breath-hold (DIBH) is a motion management technique that is commonly used in lung radiotherapy (RT) for patients with large tumor motion. Cone beam CT (CBCT) scan with DIBH is commonly used for treatment setup, and the acquisition of a CBCT scan (typically ~1 minute) requires multiple DIBHs. If a large shift of treatment couch is needed, a second CBCT scan is often acquired after the shift for verification purpose. However, the more DIBH occurred during treatment setup, the higher probability that patient may become tired and hence less able to reproduce consistent DIBH during treatment delivery, which would degrade the target coverage and deliver more dose to critical organs and normal tissues. To reduce the number of DIBHs required for treatment setup, we propose a deep learning method to locate tumor 3D location from two orthogonal and simultaneously acquired 2D projections (one is kV and the other is MV) for fast verification, which will eliminate the need for the second CBCT scan. Our proposed method, named center-ness matching network, is capable of collecting feature map that represent the probability distribution of tumor, called center-ness map, from 2D projections and re-aligning them to their projections’ angle in the Cartesian coordinate system. The tumor depth information is also collected. The center-ness matching network then re-transforms the 2D center-ness maps to the 3D center-ness map via depth learning. Finally, the 3D location of tumor is estimated from the 3D center-ness map. We conducted a retrospective study on 10 patient cases, who had undergone 4D-CT scans for motion evaluation at CT simulation and received lung RT treatments in our institution. For each 3D CT image set of a breathing phase, we simulated its 2D kV projection at gantry angle 0° and 2D MV projection at 90°. Patient specific 3D CT images of 9 phases and the corresponding 2D projections data were used for training, with the remaining phase used for testing. The target error (TE), which is of the difference in distance between the center-of-mass of ground truth tumor of 3D CT and the predicted tumor 3D location, achieved by our method was 2.6±0.7 mm. These results demonstrated the feasibility and efficacy of our proposed method, which provides a potential solution for fast verification of treatment setup of DIBH lung RT.

Automatic segmentation of the Parkinson’s disease-related tissue, the substantia nigra (SN), is an important step towards the accurate computer-aided diagnosis systems. Recently deep learning methods have achieved the state-of-the-art performance of the automated segmentation in various scenarios of medical image analysis. However, to acquire high resolution segmentation results, the conventional deep learning frameworks depend heavily on the full size of annotated data, which is pretty time-consuming and expensive for the training of the model. Moreover, the SN structure is usually tiny and sensitive to the progression of Parkinson’s disease (PD), which brings more anatomic variations among cases. To deal with these problems, this paper combines the cascaded fully convolutional network (FCN) and the size-reweighted loss function to automatically segment the tiny subcortical tissue SN from T2 MRI volumes. Different from the conventional one-stage FCNs, we cascade two FCNs in a coarse-to-fine fashion for the high resolution segmentation of the SN. The first FCN is trained to locate the SN-contained ROI and produce a coarse segmentation mask from a down-sampled MRI volume. The second FCN solely segments the SN at full resolution based on the results of the first FCN. Additionally, by giving higher weights to the SN region, the size-reweighted loss function encourages the model to concentrate on the tiny SN structure. Our results showed that the proposed FCN achieves mean dice score of 68.92% in comparison with the baseline model 66.40%.

Examination of head shape during the fetal period is an important task to evaluate head growth and to diagnose fetal abnormalities. Traditional clinical practice frequently relies on the estimation of head circumference (HC) from 2D ultrasound (US) images by manually fitting an ellipse to the fetal skull. However, this process tends to be prone to observer variability, and therefore, automatic approaches for HC delineation can bring added value for clinical practice. In this paper, an automatic method to accurately delineate the fetal head in US images is proposed. The proposed method is divided into two stages: (i) head delineation through a regression convolutional neural network (CNN) that estimates a gaussian-like map of the head contour; and (ii) robust ellipse fitting using a registration-based approach that combines the random sample consensus (RANSAC) and iterative closest point (ICP) algorithms. The proposed method was applied to the HC18 Challenge dataset, which contains 999 training and 335 testing images. Experiments showed that the proposed strategy achieved a mean average difference of -0.11 ± 2.67 mm and a Dice coefficient of 97.95 ± 1.12% against manual annotation, outperforming other approaches in the literature. The obtained results showed the effectiveness of the proposed method for HC delineation, suggesting its potential to be used in clinical practice for head shape assessment.

Accurate detection and segmentation of diffuse large B-cell lymphoma (DLBCL) from PET images has important implications for estimation of total metabolic tumor volume, radiomics analysis, surgical intervention and radiotherapy. Manual segmentation of tumors in whole-body PET images is time-consuming, labor-intensive and operator-dependent. In this work, we develop and validate a fast and efficient three-step cascaded deep learning model for automated detection and segmentation of DLBCL tumors from PET images. As compared to a single end-to-end network for segmentation of tumors in whole-body PET images, our three-step model is more effective (improves 3D Dice score from 58.9% to 78.1%) since each of its specialized modules, namely the slice classifier, the tumor detector and the tumor segmentor, can be trained independently to a high degree of skill to carry out a specific task, rather than a single network with suboptimal performance on overall segmentation.

Histopathology has played an essential role in cancer diagnosis. With the rapid advances in convolutional neural networks (CNN), various CNN-based automated pathological image segmentation approaches have been developed in computer-assisted pathological image analysis. In the past few years, Transformer neural networks (Transformer) have shown the unique merit of capturing the global long-distance dependencies across the entire image as a new deep learning paradigm. Such merit is appealing for exploring spatially heterogeneous pathological images. However, there have been very few, if any, studies that have systematically evaluated the current Transformer-based approaches in pathological image segmentation. To assess the performance of Transformer segmentation models on whole slide images (WSI), we quantitatively evaluated six prevalent transformer-based models on tumor segmentation, using the widely used PAIP liver histopathological dataset. For a more comprehensive analysis, we also compare the transformer-based models with six major traditional CNN-based models. The results show that the Transformer-based models exhibit a general superior performance over the CNN-based models. In particular, Segmenter, Swin-Transformer and TransUNet–all transformer-based–came out as the best performers among the twelve evaluated models.

Lung segmentation in dynamic thoracic magnetic resonance imaging (dMRI) is a critical step for quantitative analysis of thoracic structure and function in patients with respiratory disorders. Some semi-automatic and automatic lung segmentation methods based on traditional image processing models have been proposed mainly for CT with good performance. However, the low efficiency and robustness of these methods and inapplicability to dMRI make them unsuitable to segment the large numbers of dMRI datasets. In this paper, we present a novel automatic lung segmentation approach for dMRI based on two-stage convolutional neural networks (CNNs). In the first stage, we utilize the modified min-max normalization method to pre-process MRI for increasing the contrast between the lung and surrounding tissue and propose a corner-points and CNN based region of interest (ROI) detection strategy to extract the lung ROI from sagittal dMRI slices, which can reduce the negative influence of tissues located far away from the lung. In the second stage, we input the adjacent ROIs of target slices into the modified 2D U-Net to segment the lung tissue. The qualitative and quantitative results demonstrate that our approach achieves high accuracy and stability in terms of lung segmentation for dMRI.

e propose a fast and robust multi-class deep learning framework for segmenting COVID-19 lesions: Ground Glass opacities and High opacities (including consolidations and pleural effusion), from non-contrast CT scans using convolutional Long Short-Term Memory network for self-attention. Our method allows rapid quantification of pneumonia burden from CT with performance equivalent to expert readers. The mean dice score across 5 folds was 0.8776 with a standard deviation of 0.0095. A low standard deviation between results from each fold indicate the models were trained equally good regardless of the training fold. The cumulative per-patient mean dice score (0.8775±0.075) for N=167 patients, after concatenation, is consistent with the results from each of the 5 folds. We obtained excellent Pearson correlation (expert vs. automatic) of 0.9396 (p<0.0001) and 0.9843 (p<0.0001) between ground-glass opacity and high opacity volumes, respectively. Our model outperforms Unet2d (p<0.05) and Unet3d (p<0.05) in segmenting high opacities, has comparable performance with Unet2d in segmenting ground-glass opacities, and significantly outperforms Unet3d (p<0.0001) in segmenting ground-glass opacities. Our model performs faster on CPU and GPU when compared to Unet2d and Unet3d. For same number of input slices, our model consumed 0.83x and 0.26x the memory consumed by Unet2d and Unet3d.

Purpose: The aim of this work is to automatically detect and estimate the centerline of the pancreatic duct accurately. The proposed method uses four different algorithms for tracking the pancreatic duct in each of four type pancreatic zones. Method: The pancreatic duct was divided into 4 zones; Zone A has a clearly delineated pancreatic duct, Zone B is obscured, Zone C runs from visible segment to the pancreas’ tail and Zone D extends from head of the pancreas to the first visible point. The pancreatic duct is obscured in regions of lengths from 10-40 mm. Proposed method combines deep learning CNN for duct segmentation, followed by Dijkstra's rooting algorithm for estimation of centerline in Zones A and Zones B. In Zone C and D, the centerline was estimated using geometric information. The reference standard for the pancreatic duct was determined using non-obscured data by skilled technologists. Results: Zones A, which used a neural network method, had a success rate of 94％. In Zone B, the difference was <3mm when obscured interval was 10-40mm In Zone C and D, distance between computer estimated pancreas head and tail points and operator determined anatomical point was 10mm and 19mm, respectively. Optimal characteristic cost functions for each zone allow the natural centerline to be estimated even in obscured region. The new algorithms increased the average visible centerline length by 146% with calculation time of <40 seconds.

Deep learning models have obtained state-of-the-art results for medical image analysis. However, CNNs require a massive amount of labelled data to achieve a high performance. Moreover, many supervised learning approaches assume that the training/source dataset and test/target dataset follow the same probability distribution. Nevertheless, this assumption is hardly satisfied in real-world data and when the models are tested on an unseen domain there is a significant performance degradation. In this work, we present an unsupervised Cross-Modality Adversarial Domain Adaptation (C-MADA) framework for medical image segmentation. C-MADA implements an image-level and feature-level adaptation method in a two-step sequential manner. First, images from the source domain are translated to the target domain through an unpaired image-to-image adversarial translation with cycle-consistency loss. Then, a U-Net network is trained with the mapped source domain images and target domain images in an adversarial manner to learn domain-invariant feature representations and produce segmentations for the target domain. Furthermore, to improve the network’s segmentation performance, information about the shape, texture, and contour of the predicted segmentation is included during the adversarial training. C-MADA is tested on the task of brain MRI segmentation from the crossMoDa Grand Challenge and is ranked within the top 15 submissions of the challenge.

MR-guided laser interstitial thermal therapy (LITT) has been regarded as a promising treatment option in neurosurgical oncology. One major advantage of MR-guided LITT is that temperature measurement can be provided to monitor the ablation in nearly real-time. The intraoperative temperature images make it possible to estimate thermal damage during the ablation. Several thermal damage models have been developed and used in LITT procedure, but the model parameters were mainly derived from laboratories in vitro experiments and varied in a wide range. As the frequency of MR-guided LITT procedure increases, there is a growing need to optimize thermal damage models utilizing clinical data. In this study, we dug into the relationship between the temperature history and postoperative ablation image collected in brain LITT surgery. Compared with the previous studies where the thermal damage models could only predict two tissue states: normal and ablated, our study considered three tissue states for thermal damage prediction: ablated, transitional and normal. We noticed that for different tissue states, the damage values calculated according to the Arrhenius formula were in different distribution patterns. Therefore, we developed a distribution-based thermal damage model. MR images of 15 patients were used in our study. The results showed that the proposed method could predict three tissue states and had higher prediction accuracy than the traditional Arrhenius model in clinical application.

We propose a pattern-expression method based on rank-one tensor decomposition for analysis for substantia nigra in T2-weighted images. Capturing discriminative features in observed medical data is an important task in diagnosis. In diagnosing Parkinson’s disease, capturing the change of volumetric data of substantia nigra supports the clinical diagnosis. Furthermore, in drug discovery researches for Parkinson’s disease, statistical evaluations of changes of substantia nigra, which are caused by a developed medicine, also might be necessary. Therefore, we tackle the development of the pattern-expression method to analyse volumetric data of substantia nigra. Experimental results showed the different distributions of computed coefficients for rank-one tensors between Parkinson’s disease and healthy state. The results indicated the validity of the tensor-decomposition-based pattern-expression method for the analysis.

Type 1 diabetes (T1D) affects over 200,000 children and is associated with an increased risk of cognitive dysfunction. Prior imaging studies suggest the neurological changes underlying this risk are multifactorial, including macrostructural, microstructural, and inflammatory changes. However, these studies have yet to be integrated, limiting investigation into how these phenomena interact. To better understand these complex mechanisms of brain injury, a well-powered, prospective, multisite, and multimodal neuroimaging study is needed. We take the first step in accomplishing this with a preliminary characterization of multisite, multimodal MRI quality, motion, and variability in pediatric T1D. We acquire structural T1 weighted (T1w) MRI, diffusion tensor MRI (DTI), functional MRI (fMRI), and magnetic resonance spectroscopy (MRS) of 5-7 participants from each of two sites. First, we assess the contrast-to-noise ratio of the T1w MRI and find no differences between sites. Second, we characterize intervolume motion in DTI and fMRI and find it to be on the subvoxel level. Third, we investigate variability in regional gray matter volumes and local gyrification indices, bundle wise DTI microstructural measures, and N-acetylaspartate to creatine ratios. We find the T1-based measures to be comparable between sites before harmonization and the DTI and MRS-based measures to be comparable after. We find a 5-15% coefficient of variation for most measures, suggesting ~150-200 participants per group on average are needed to detect a 5% difference across these modalities at 0.9 power. We conclude that multisite, multimodal neuroimaging of pediatric T1D is feasible with low motion artifact after harmonization of DTI and MRS.

Branch retinal artery occlusion (BRAO) is an ophthalmic emergency. Acute BRAO is a clinical manifestation of BRAO. Due to its various shapes, locations and the blurred boundary, the automatic segmentation of acute BRAO is very challenging. To tackle these problems, we propose a novel method based on deep learning for automatic acute BRAO segmentation in optical coherence tomography (OCT) image. In this method, a novel Bayes posterior attention network, named as BPANet, is proposed for precise segmentation of the lesion. Our major contributions include: (1) A novel Bayes posterior probability based spatial attention module is used to enhance the information of lesion region. (2) An effective max-pooling and average-pooling channel attention module is embedded into BPANet to improve the effectiveness of the feature extraction. The proposed method is evaluated on 472 OCT B-scan images with a 4-fold cross validation strategy. The mean and standard deviation of Dice similarity coefficient, true positive rate, accuracy and intersection over union are 85.48±1.75%, 88.84±1.19%, 98.63±0.48% and 76.88±2.92%, respectively. The primary results show the effectiveness of the proposed method.

Recent advances in magnetic resonance (MR) scanner quality and the rapidly improving nature of facial recognition software have necessitated the introduction of MR defacing algorithms to protect patient privacy. As a result, there are a number of MR defacing algorithms available to the neuroimaging community, with several appearing in just the last five years. These various approaches have qualities that have been explored with respect to skull stripping masks or identifiability of the patient in previous works. However, to our knowledge there has been no evaluation of the subsequent impact of these defacing algorithms on a neuroimaging pipeline. In this work, we use six MR defacing algorithms on 179 subjects from the OASIS-3 cohort and 21 subjects from the Kirby 21 dataset, then apply a neuroimaging pipeline to the resultant defaced images. We compare the consistency of the output from the pipeline using the defaced images with the output of the same pipeline without defacing the MR data.