This PDF file contains the front matter associated with SPIE Proceedings Volume 12037, including the Title Page, Copyright information, Table of Contents, and Conference Committee listings.

Limited access to medical datasets, due to regulations that protect patient data, is a major hinderance to the development of machine learning models for computer-aided diagnosis tools using medical images. Distributed learning is an alternative to training machine learning models on centrally collected data that solves data sharing issues. The main idea of distributed learning is to train models remotely at each medical center rather than collecting the data in a central database, thereby avoiding sharing data between centers and model developers. In this work, we propose a travelling model that performs distributed learning for biological brain age prediction using morphological measurements of different brain structures. We specifically investigate the impact of nonidentically distributed data between collaborators on the performance of the travelling model. Our results, based on a large dataset of 2058 magnetic resonance imaging scans, demonstrate that transferring the model weights between the centers more frequently achieves results (mean age prediction error = 5.89 years) comparable to central learning implementations (mean age prediction error = 5.93 years), which were trained using the data from all sites hosted together at a central location. Moreover, we show that our model does not suffer from catastrophic forgetting, and that data distribution is less important than the number of times that the model travels between collaborators.

Federated learning (FL) has emerged with increasing popularity in the medical image analysis field. In collaborative model training, it provides a privacy-preserving scheme by keeping data localized. In FL frameworks, instead of collecting data from clients, the server learns a global model by aggregating local training models from clients and broadcasts the updated model. However, in the situation where data is not identically and independently distributed (non-i.i.d), the model aggregation requires frequent message passing, which may face the communication bottleneck. In this paper, we propose a communication-efficient FL framework based on the adaptive server-client model transmission. The local model in the client will only be uploaded to the server under the conditions of (1) a probability threshold and (2) an informative model updating threshold. Our framework also tackles the data heterogeneity in federated networks by involving a proximal term. We evaluate our approach on a simulated multi-site medical image dataset for diabetic retinopathy (DR) rating. We demonstrate that our framework not only maintains the accuracy on non-i.i.d dataset but also provides a significant reduction in communication cost compared to other FL algorithms.

Chest Computational Tomography (CT) scans present low cost, speed and objectivity for COVID-19 diagnosis and deep learning methods have shown great promise in assisting the analysis and interpretation of these images. Most hospitals or countries can train their own models using in-house data, however empirical evidence shows that those models perform poorly when tested on new unseen cases, surfacing the need for coordinated global collaboration. Due to privacy regulations, medical data sharing between hospitals and nations is extremely difficult. We propose a GAN-augmented federated learning model, dubbed ST-FL (Style Transfer Federated Learning), for COVID-19 image segmentation. Federated learning (FL) permits a centralised model to be learned in a secure manner from heterogeneous datasets located in disparate private data silos. We demonstrate that the widely varying data quality on FL client nodes leads to a sub-optimal centralised FL model for COVID-19 chest CT image segmentation. ST-FL is a novel FL framework that is robust in the face of highly variable data quality at client nodes. The robustness is achieved by a denoising CycleGAN model at each client of the federation that maps arbitrary quality images into the same target quality, counteracting the severe data variability evident in real-world FL use-cases. Each client is provided with the target style, which is the same for all clients, and trains their own denoiser. Our qualitative and quantitative results suggest that this FL model performs comparably to, and in some cases better than, a model that has centralised access to all the training data.

Respiratory rate is one of the most important but underutilized vital signs. Modern medical practices have standard automated methods to detect abnormalities in heart rate, blood oxygen concentration, temperature, and blood pressure, but not for respiratory rate. Due to the time-consuming, patient-contact intensive observations that are required to measure respiratory rate is tedious to do in a hospital setting.

This paper proposes a method to measure respiratory rate using a convolutional neural network in thermal imaging. Twenty subjects were used to collect 38,394 images of inhaling and exhaling used for training the convolutional neural network. The convolutional neural network has proved to be highly accurate when tested on ten different subjects, each in three distinct positions. In the 30 validation recordings, 100% of respiratory rate values were within ±2 breaths of the manually counted value. If the performance is measured at ±1 breath, then the accuracy of this respiratory rate detector is 86.6% which is more accurate than manual counting. It has also been demonstrated that subjects at slight angles (10° - 45°) between the camera and the subject’s midsagittal plane provide a greater number of exact breath counts.

This method does not require patient contact and supplies real time operation which can improve hospital efficacy by allowing medical professionals to perform other duties while measurements are being taken. The results of this study provide promise to provide hospitals with an effective method for the future of respiratory rate detection using convolutional neural network in thermal imaging.

Retinopathy of prematurity (ROP) is a retinal vascular disease that affects premature infants and can result in blindness within days if not monitored and treated. A disease stage for increased scrutiny and treatment within ROP is “plus disease,” characterized by increased tortuosity and dilation of posterior retinal blood vessels. Monitoring of ROP occurs with routine imaging, typically using expensive instruments ranging from $50-140K. In low-resource areas of the world, smartphone cameras and inexpensive Volk 28D lenses are being used to image the fundus, albeit with lower fields of view and image quality than the expensive systems. We developed a preprocessing pipeline to enhance vessel visualization and harmonize images for automated analysis using deep learning algorithms. After preprocessing, vessel contrast was enhanced by 90% as assessed by the contrast improvement index. In an image quality evaluation, 441 images were evaluated by pediatric ophthalmologists from the US and South America, all with years of experience diagnosing ROP and plus disease. 100% of participating ophthalmologists either agreed or strongly agreed that vessel visibility was improved in the processed images. A preliminary deep learning binary classifier (plus vs. no plus disease) was developed using GoogLeNet. Using smartphone images harmonized via preprocessing (e.g., vessel enhancement and size normalization) and augmented in physically reasonable ways (e.g., image rotation), we achieved an exceptional accuracy of 0.96 for plus disease on a limited dataset. These promising results suggest the potential to create algorithms and software to improve usage of cell phone images for ROP staging.

A person spends a significant portion of time driving a vehicle. This time serves several applications, such as unobtrusive health monitoring with sensors that are mounted inside the car. Such a car can perform regular medical checkups or other tasks such as drunk driver detection. For such tasks, driver behavior monitoring is essential. Several approaches utilize data from different modalities and sensors. Video-based recognition is used increasingly and usually combined with deep learning. In this work, we propose an end-to-end transfer learning approach using temporal pyramidal networks (TPN’s) on top of a ResNet-50 backbone that is pre-trained on the Kinetics400 dataset. We further perform a comparative analysis with the inflated 3D ConvNet network (I3D). We aim to boost training efficiency while improving accuracy as compared to previous work. The extracted videos from the DriveAct dataset have been captured from a single near-infrared (NIR) camera mounted on the rear-view mirror. Using these videos for training and evaluation, we achieve the best validation accuracy of 75.74%. This work has several potentials to be extended, generalizing to a multi-camera setup and combining multi-modal data to increase accuracy significantly. It further serves as a baseline for in-car health monitoring.

Face detection in videos from smart cars or homes is becoming increasingly important in human-computer interaction, emotion recognition, gender and age identification, driving assistance, and vital sign measurements, as heart rate and respiratory rate is derived from the video. However, face detection suffers from variations in illumination, subject motion, different skin colors, or camera distances. We compare three algorithms for in-car application: Haar cascade classifier (HCC), histogram of oriented gradients (HoG), and a deep neural network (DNN). For evaluation, we consider the freely available “driver monitoring dataset” multimodal database (DMD) and self-collected videos recorded in a research car. We analyze run-time, accuracy, and F1-score. HoG has highest computational time as compared to HCC and DNN with 2.99 frames per second(fps), 7.00 fps, and 18.25 fps, respectively. For DMD, the F1-scores are 91.75%, 95.91%, and 99.48% for HCC, HoG, and DNN respectively, and 88.05%, 83.68%, and 99.66% for our dataset, respectively. All in all, DNN is fastest and most accurate. Moreover, we observed that DNN handles occlusions and varying illumination better than the other approaches. In conclusion, DNN can be applied successfully for in-car face detection as a first step towards real-time continuous health monitoring.

Road traffic crashes have become the leading cause of death for young people. Approximately 1.3 million people die due to road traffic crashes, and more than 30 million people suffer non-fatal injuries. Various studies have shown that emotions influence driving performance. In this work, we focus on frame-level video-based categorical emotion recognition in drivers. We propose a Convolutional Bidirectional Long Short-Term Memory Neural Network (CBiLSTM) architecture to capture the spatio-temporal features of the video data effectively. For this, the facial videos of drivers are obtained from two publicly available datasets, namely Keimyung University Facial Expression of Drivers (KMU-FED), a subset of the Driver Monitoring Dataset (DMD), and an experimental dataset. Firstly, we extract the face region from the video frames using the Facial Alignment Network (FAN). Secondly, these face regions are encoded using a lightweight SqueezeNet CNN model. The output of the CNN model is fed into a two-layered BiLSTM network for spatio-temporal feature learning. Finally, a fully-connected layer outputs the emotion class softmax probabilities. Furthermore, we enable interpretable visualizations of the results using Axiom-based Grad-CAM (XGrad-CAM). For this study, we manually annotated the DMD and our experimental dataset using an interactive annotation tool. Our framework achieves an F1-score of 0.958 on the KMU-FED dataset. We evaluate our model using Leave-One-Out Cross-Validation (LOOCV) for the DMD and the experimental dataset and achieve average F1-scores of 0.745 and 0.414 respectively.

Human activities play a vital role in many health-related fields of research. For example, changes in activities of daily living can predict neurodegenerative diseases like Alzheimer’s disease or the risk of suffering from a fall. Therefore, automatically recognizing activities in videos has become of scientific interest. In contrast to wearable devices, video cameras have the advantage of being truly unobtrusive without any physical contact, continuous as they do not require charging, and cannot be forgotten to wear. This work proposes a novel approach for multi-camera and multi-person human activity recognition (HAR) in videos. The aim is to classify each person in each frame of a video in one of the five classes: “Lying”, “Sitting”, “Standing”, “Walking” or “Falling”. We use a combination of YOLOv4 (person detection), DeepSort (tracking), a convolutional neural network (CNN, features extraction), and an attention-based multi-layer long short-term memory network (classification) to track actions of multiple subjects. Our entire dataset comprises of four publicly-available datasets (ETRI - Activity 3D, MSR Daily Activity 3D, HAR-UP Fall Dataset, High-Quality Fall Simulation Data) with a total file size of 300 gigabytes. In our experiments, we pick random subsets of 1% and 0.25% of data for training and testing respectively. We achieve a classification accuracy of 95%. In the presence of a single subject in the room, predictions from multiple cameras are combined using soft voting, which further improves the accuracy. In summary, HAR in videos is feasible using the proposed combination of machine learning techniques.

Recognizing workflow in the operating room (OR) is essential for developing context-aware systems to provide rich information to enhance patient outcomes by providing performance feedback to the staff post operation or by managing OR scheduling and utilization in real-time. However, manual event reporting can be unreliable and time-consuming. Therefore, our work aims to develop an automatic OR activity analysis framework for recognizing OR workflow for efficiency understanding, analysis and optimization. Such tasks are challenging due to the OR staff wearing similar scrubs and face masks in this setting. The presence of large machinery and equipment also occlude much camera views. By utilizing multiple pre-calibrated cameras mounted in the OR, our system localizes, tracks and recognizes the activity of each staff in OR. First, multi-person localization is done in 2D by using a top-down neural network. Next, an efficient and novel 3D reconstruction by tracking scheme is employed by matching 2D poses across different camera views with back-projected 2D poses from previous frame. Lastly, actions, such as standing, walking, sitting, crouching and lying, are recognized via prior pose knowledge and a pose graph convolution network. We evaluated pose estimation accuracy using percentage of correct key points (PCK) metric and achieved 78.0% accuracy on MVOR dataset, which outperformed state-of-art performance (76.2%). Based on these poses, the actions recognized are 96% accurate. Given the results, our proposed framework for automatic OR activity analysis demonstrates its potential to facilitate further online or offline workflow analysis.

The purpose of this project is to prepare image data set to develop AI systems to serve screening and diagnosis of breast cancer research field. Whereas early detection could have a positive impact on decreasing mortality, as this could offer more options for successful intervention and therapies to reduce the chance of malignant and metastatic progression. Six students, one research technologist, and one consultant in radiology collected the images and the patients’ information. The images extracted from three imaging modalities: the Hologic 3D Mammography, Philips and Super Sonic ultrasound Machines, and GE and Philips machines for MRI. The cases were graded by a trained radiologist. A total of 3085 DICOM format images have collected for the period between 2008 – 2020 for 890 females patients ages 18 to 85. The largest portion in the data is dedicated for mammograms (51.3%), and then ultrasound (31.7%), and MRI exams (17%). There were 593 malignant cases while the benign cases were 2492 cases. The diagnosis was confirmed by biopsy technique after mammogram and ultrasound exams. The data will be continually collected in the future to serve the artificial intelligence research field and the public health community. The updated information about the data will be available on: https://kaimrc.med.sa/?page_id=11767072

The purpose of this project is to collect a large ocular images dataset to be used to develop Artificial Intelligence (AI) systems to server an early detection for any ocular diseases. The aim is to decrease blindness rates and promoting better vision quality. The developed systems will contribute in early disease detection and maximizing the accuracy of diagnosis and treatment decisions and reduces the burdens on medical professionals. Two different Optical Coherence Tomography (OCT) machines at the retina and glaucoma clinics in King Abdulaziz Medical City in Riyadh, Saudi Arabia were used to collect the images and the associated clinical information. A total of 58114 retinal images for 4863 patients, 2394 are male and 2469 are female, for the age between 4 to 99 years old captured between the period of 2007- 2021. whereas 40722 images were extracted from Heidelberg Engineering OCT and 17392 images extracted from Topcon OCT. The images will be visually and manually labeled and annotated respectively by multiple specialized ophthalmologists. The developed AI systems will serves the population and health care system in early detection of preventable blinding diseases. Moreover, these systems will reduce the high cost of the late ocular diseases detections. The updated information regarding the data will be available through the link: https://kaimrc.med.sa/?page_id=11767072

Oral cavity cancer is a common cancer that can result in breathing, swallowing, drinking, eating problems as well as speech impairment, and there is high mortality for the advanced stage. Its diagnosis is confirmed through histopathology. It is of critical importance to determine the need for biopsy and identify the correct location. Deep learning has demonstrated great promise/success in several image-based medical screening/diagnostic applications. However, automated visual evaluation of oral cavity lesions has received limited attention in the literature. Since the disease can occur in different parts of the oral cavity, a first step is to identify the images of different anatomical sites. We automatically generate labels for six sites which will help in lesion detection in a subsequent analytical module. We apply a recently proposed network called ResNeSt that incorporates channel-wise attention with multi-path representation and demonstrate high performance on the test set. The average F1-score for all classes and accuracy are both 0.96. Moreover, we provide a detailed discussion on class activation maps obtained from both correct and incorrect predictions to analyze algorithm behavior. The highlighted regions in the class activation maps generally correlate considerably well with the region of interest perceived and expected by expert human observers. The insights and knowledge gained from the analysis are helpful in not only algorithm improvement, but also aiding the development of the other key components in the process of computer assisted oral cancer screening.

With the rapid growth in deep learning research for medical applications, the value of making these techniques accessible to clinics also increases. Many medical technology companies now offer deep learning contouring, but researchers are usually limited to the proprietary pre-trained models. To fully explore the technology, researchers must build deep learning pipelines from scratch. We developed an open-source framework for producing automatic contours for 11 common organs-at-risk (OAR) for head and neck planning CT studies using a convolutional neural network (CNN). The pipeline handles DICOM file ingestion, data pre-processing, CNN utilization, output postprocessing, and DICOM structure set file creation to allow end-to-end use interfacing directly with DICOM files. We trained a standard U-Net model on 210 anonymized head and neck patients from our clinic, validated the model’s performance on a test set of 19 patients, and provide the pre-trained weights as a part of the pipeline offering to allow for immediate use. Scripts for retraining the model are also provided to allow customization and new research efforts. Additionally, we offer a framework of all necessary files to support browser-based, no-code contour generation using the Flask package for Python. These contributions lay the foundation for clinical workflow integration. All files are freely available in a public GitHub repository (https://github.com/jasbach/HN_UNet_Autosegmentation_Tool) and are ready for immediate use. Our work offers a demonstrably successful deep learning tool for automatic contouring with a reduced barrier to entry for novice personnel wishing to expand their efforts into the discipline.

Primary angle closure disease (PACD) is a leading cause of permanent vision loss worldwide, so early treatment of patients suffering from symptoms of PACD is crucial to prevent vision loss. Gonioscopy is the current clinical standard for diagnosing PACD. However, gonioscopy is a qualitative subjective assessment method. Thus, there is a need for a quantitative method to diagnose PACD. Anterior Segment Optical Coherence Tomography (AS-OCT) is an imaging modality which produces images of anterior structures such as the anterior chamber angle. Adoption of AS-OCT has been slow due to AS-OCT analysis not being standardized and inefficient. Currently, users must annotate each image by hand using proprietary software and use expert knowledge to diagnose PACD based on the key features annotated. Using an imaging-informatics based approach on a dataset of over 900 images we have developed a system to streamline and standardize AS-OCT analysis. This system will be DICOM compatible to promote standardization of AS-OCT images. This system will be attached to a HIPAA compliant database and will require a secure login to protect patient privacy. We have developed a streamlined approach towards annotating key features in AS-OCT images which will be used to validate the results of an automatic segmentation method. The automatic segmentation method will be integrated into the system to increase the efficiency of analyzing AS-OCT images and eliminate the need to annotate images for clinical diagnosis. These features will be used in the future to classify PACD based on the severity of the angle closure.

Pneumoconiosis is an occupational respiratory disease caused by inhaling dust into the lungs. In Japan, 240,000 people undergo pneumoconiosis screening every year. X-rays are used worldwide to classify the severity of pneumoconiosis. It is important to distinguish between stage 0/1 and stage 1/0, which are eligible for recognition of occupational injury. CT images are expected to provide more accurate diagnosis because they can be confirmed in three dimensions compared to X-rays. We extract micro-nodules from 3D CT images for each severity of pneumoconiosis, and analyze and evaluate the number, size, and CT values of micro-nodules in each lung lobe.

Machine-learning-based solutions need sufficient manually labeled training data to produce accurate predictions, which can hinder their performance for rare diseases with limited data. We show how to use a newly developed algebraic topology-based machine learning method that analyzes the visual pattern of the data to accurately predict hepatic decompensation in patients with Primary Sclerosing Cholangitis.

The results demonstrate that the proposed methodology discriminates between Early Decompensation and Not Early groups. We found that the algebraic topology-based machine-learning approach allows us to make accurate predictions from small datasets such as predicting early and not early hepatic decompensation.

Purpose: Patient privacy rules require removal of Protected Health Information (PHI) before sharing images publicly. Manual de-identification is no longer scalable due to the rapid increase in imaging data volume. Our goal was to configure and test the efficacy of an automated medical image de-identification (MIDI) pipeline using cloud services.

Materials and Methods: Training and test datasets for validation of image de-identification, specifically prepared by placement of synthetic PHI in DICOM headers and image pixel data, were prepared by The Cancer Imaging Archive (TCIA). These datasets included 1,836/14,372 images from 21/93 patients, respectively. Answer keys based on TCIA de-identification conventions were made available for the two datasets. The MIDI pipeline was configured using the Google Cloud Platform Healthcare API, which is based on Google’s Data Loss Prevention API for sensitive information detection. Performance was also measured by monitoring throughput.

Results: For DICOM header data elements, 99.8% of expected actions were performed correctly. The two incorrect actions included one false-positive case (information removed incorrectly), and one false-negative case (PHI not removed). For the image pixel data, one false-positive was noted. There were no false negatives; all sensitive information was correctly removed from all image pixel data. Throughput averaged at 58.4 images per second.

Conclusion: The current implementation of the MIDI pipeline holds great promise for automated de-identification at scale. However, verification by a human expert is currently recommended. Optimization of the underlying algorithm could further increase accuracy.

We have created a database for the storage of DICOM data using MongoDB. Data is stored in chunks using gridfs and accessible via queries on any field of the DICOM header, as well as other pre-defined metadata fields. When data is imported into the database, a custom Python plugin extracts metadata from the DICOM header for each file, which is used to populate the database. In addition to the UID values stored in the DICOM header, an MD5 checksum is computed for each file and referenced in the database to ensure that data is not duplicated. The metadata is tracked using a local instance of MongoDB Charts.

Data is replicated across two NVMe servers connected with 100 Gbps ethernet for fast replication. The two servers are configured as a MongoDB replica set with an arbiter running on a third server that is responsible for promoting either of the data-bearing nodes to be the primary access point based on usage and availability. The database is accessible via a custom PyTorch connector that connects the database to the 100 Gbps network for access by a HPC cluster for deep learning applications and via a web interface on the local network used to pull or upload data.

The database is being utilized for the storage of SPECT Medical Imaging data. The database improves our ability to catalog collaboration data, identify data that is relevant for deep learning studies, and to perform data mining on the existing dataset.

Sports medicine research is a developing field which uses a processed combination of videos, force data, and accelerometer data. Currently raw data is transferred onto hard drives and then manually processed by researchers. However, there is currently no standard method to store the raw, unprocessed data collected by the various sports sensors. These sensors lack the capability to store raw data themselves, so data must be stored elsewhere after collection. Once this data is stored there is no streamlined way to tag or retrieve this data. Sports data is also often shared between collaborating institutions. Institutions need a way to centrally store data to facilitate easier transfer between institutions. In a similar fashion, imaging modalities acquire raw imaging data that is currently stored directly in the modality systems. Therefore, there is a need for a standardized, streamlined system which can both store the raw, unprocessed data and can tag that raw data to allow for more efficient data query. Using an imaging-informatics based approach we have developed a “will-call” data mart which allows users to store raw sports data along with tags. This data mart will allow users to quickly query and retrieve data. The data mart’s efficacy will be measured by testing expert and novice users on the speed and ease of using the data mart and comparing it to the traditional method of data storage. In the future this data mart can be integrated into other sports related systems such as the integrated biomechanics informatics system (IBIS).

The field of biomechanics integrates a variety of data types to provide meaningful feedback to athletes, coaches, and healthcare professionals to reduce injury and improve performance. These different sources of data include waveform, video, discrete, and performance metrics. An integrated biomechanics informatics system (IBIS)^1,2 has been developed to apply imaging informatics principles to the field of biomechanics. The IBIS system helps create a space for stakeholders to make informed decisions using multimodal biomechanics data. This current work details the development of two new decision support tools within the IBIS system that enhance the ability of users to effectively analyze and interpret data. These two tools include an R Shiny statistical analysis tool and an algorithm for detecting foot contact with the ground. Organizing data into SQL databases on the IBIS system provides the backend for the R Shiny analysis tools. This connection provides statistical analysis tools to review data over time and across individuals. Its integration also incorporates interactive data visualization techniques to put biomechanical data in context. A semi-automated foot contact detection algorithm was created to reduce the need for researchers to manually search through thousands of video frames to synchronize video data accurately with other information such as force data. Overall, these two improvements to the system will support researchers working to provide meaningful insight about human performance and injury prevention. These additions to the IBIS system provide quicker access to data, develop standardized pipelines, and create additional time for biomechanists to focus on interpreting results rather than data management.

Despite its recent introduction into clinical medical practice, three-dimensional (3D) printing is having a significant impact on patient care by generating patient-specific 3D anatomic models and surgical guides from volumetric medical imaging data sets (e.g., high-resolution computed tomography (CT), magnetic resonance imaging (MRI), etc). These models are transforming care by providing new but not limited to methods of data visualization, patient-specific prostheses and surgical guides. However, there is a lack of standardization, particularly in terms of quality assurance (QA) to ensure that the 3D printed model accurately represents the corresponding patient’s anatomy and pathology. This becomes even more problematic given multiple printing technologies, variability across vendors, and inter printer capabilities, all of which have an impact on printing accuracy and precision. In this study, we investigated printing accuracy on a diverse selection of 3D printers commonly used in the medical field. A specially designed 3D printing QA phantom was periodically printed on 16 printers used in our practice, covering five distinct printing technologies and eight different vendors. Longitudinal data were acquired over a period of six months with the QA phantom printed monthly on each printer. Qualitative assessment and quantitative measurements were obtained for each printed phantom. The accuracy and precision were assessed by comparing quantitative measurements with reference values of the phantom. Data were then compared among printer models, vendors, and printing technologies. Longitudinal trends were also analyzed. Results revealed differences in accuracy across printers. It was found that material jetting and vat photopolymerization printers were most accurate. Printers using the same 3D printing technology but from different vendors also showed differences in accuracy, most notably between vat photopolymerization printers from two different vendors. Furthermore, differences in accuracy were found between printers from the same vendor using the same printing technology, but different models/generations. These results show how factors such as printing technology, vendor, and model can impact 3D printing accuracy, which should be appropriately considered in practice to avoid potential detrimental consequences.

We developed a 3D-image-based unsupervised prediction model, called vox2pred, for predicting the progression of pulmonary diseases based on a conditional generative adversarial network (cGAN). The architecture of the vox2pred model includes a time generator that consists of an encoding convolutional network and a fully connected prediction network, and a discriminator network. The time generator is trained to generate the progression time from the chest 3D CT image volumes of each patient. The discriminator is a patch-based 3D-convolutional network that is trained to differentiate between “predicted pairs” of a chest CT image volume and a predicted progression time from “true pairs” of the chest CT image volume and the corresponding observed progression time of the patient. For a pilot evaluation, we retrospectively collected high-resolution chest CT images of 141 patients with the coronavirus disease 2019 (COVID-19). The progression predictions of the vox2pred model on these patients were compared with those of existing clinical prognostic biomarkers by use of a two-sided t-test with bootstrapping. Concordance index (C-index) and relative absolute error (RAE) were used as measures of the prediction performance. The bootstrap evaluation yielded C-index and RAE values of 87.4% and 18.5% for the vox2pred model, whereas those for the visual assessment of the CT images in terms of a total severity score were 62.4% and 51.8%, and for the total severity score for crazy paving and consolidation (CPC), they were 64.7% and 51.3%, respectively. The increase in the accuracy of the progression prediction by the vox2pred model was statistically significant (p < 0.0001), indicating the high effectiveness of vox2pred as a prediction model for pulmonary disease progression in chest CT.

This contribution presents a streamlined data pipeline to bring medical 3D scans onto Augmented Reality (AR) hardware. When a 3D scan is visualized on a 2D screen, depth information is lost and doctors have to rely on their experience to map the displayed data to the patient. Showing such a scan in AR addresses this problem, as one can view that scan in real 3D. To achieve this, the scan produced by a medical scanner has to be preprocessed by the user and brought onto the AR hardware. Usually, many manual steps are involved in achieving this, which require technical knowledge about the underlying software and hardware components and impede acceptance of this new technology by the target group, medical personnel. This work presents a streamlined pipeline for this process, leading to an enhanced user experience. The core component of the pipeline is a web application, to which a user can upload the direct output of a medical scanner. The scan can be interactively segmented by the user, after which both the scan and segment are stored on a server. Additionally, this paper introduces an AR application, which can be used to browse through patients and view their scans and previously created segments. We evaluate our streamlined data pipeline and AR application in a user study, reporting the results of a system usability questionnaire and a Thinking Aloud test.

Spatial alignment of longitudinal images has been shown to improve the ability of radiologists to evaluate anatomical changes over time. However, although automated alignment using image registration algorithms is routinely performed in research applications, it is not as common within clinical workflows, partly due to limitations in the capabilities of many PACS viewing systems, as well as concerns regarding modification of the original image data. In this work, we propose an approach to integrate spatial registration into radiology workflows without altering the original pixel intensities of the acquired data. Baseline images and follow-up images are received from the PACS, co-registered within a standard reference space, and sent back to the PACS as a separate study. A key advantage of our approach is that the pixel data is not interpolated or modified in any way. The alignment of images is specified solely through modification of DICOM tags, allowing multiplanar reformation features of the PACS viewer to perform the spatial transformations. By leaving the imaging data intact, artifacts that may result from multiple transformations or oscillating interpolating functions are minimized. We demonstrate the use of this approach on magnetic resonance images of the brain, although the framework is applicable to any type of three-dimensional radiological imaging.

This study aims to display the efficient generation of three-dimensional (3D) models of medical scans and the ability for physicians to utilize the models via augmented reality (AR) head mount displays. The ability to view and interact with 3D models of patients’ medical scans on head mount displays (HMD) such as the Microsoft HoloLens 2, opens a wide range of new possibilities for more accurate and intuitive ways for physicians to approach preoperative and intraoperative planning. Traditionally, the manual workflow for generating AR models of medical scans involves various software packages that are required to manually carry out steps such as image segmentation, mesh refinement, and file conversion¹. Our web-based application automates the steps involved in generating AR models with end-to-end integration from image uploading to viewing and annotating the 3D model collaboratively on multiple AR headsets simultaneously.

In addition to the main functions involving automated segmentation, interpolation, resizing, and the cropping of uploaded DICOM images, users can now automatically convert the files into a point cloud (PLY), which can be viewed and interacted with through a preview screen implemented onto the web app. Furthermore, these 3D models can be directly uploaded to an AR headset to be viewed and annotated by multiple AR headsets simultaneously using the AR app developed for this workflow.

Over 50,000 people die every year from colorectal cancer in the United States. Early detection and removal of the types of benign pre-cancerous polyps that can develop into cancers would largely prevent these deaths. Non-cathartic laxative-free computed tomographic (CT) colonography has been shown to provide an effective complete colorectal examination that is easy to tolerate by patients. Instead of physical bowel cleansing, a method called electronic cleansing (EC) is used to perform a virtual cleansing of the colon on the acquired CT colonography images. In this preliminary study, we investigated the possibility of using 3D generative adversarial network (GAN) based unpaired contrastive learning to perform EC in laxative-free CT colonography. The unpaired training samples were collected from clinical laxative-free and cathartically cleansed CT colonography cases. The evaluation was performed by testing the model with a number of clinical laxative-free CT colonography cases. Our preliminary results indicate that the 3D GAN-based model was able to learn to perform EC in laxative-free CT colonography. However, we also identified some problems that need to be addressed before the approach can be considered mature enough for clinical application.

Serrated polyps have emerged as a new important target lesion in colorectal cancer screening. Although artificial intelligence (AI) can be used to detect serrated polyps at a high accuracy in CT colonography, it is also important to understand the uncertainties regarding the decisions made by the AI. In this pilot study, we explored the quantification of the uncertainty in 3D deep learning for the detection of serrated polyps in CT colonography. The uncertainty was estimated by use of a Monte-Carlo dropout method, and quantified by characterizing the variance of the predictions made on the Monte-Carlo samples. For a preliminary evaluation, we performed a 10-fold per-patient cross-validation to compare the accuracies and uncertainties of the detections made by the two 3D DenseNet models that we previously identified as having a high performance in the detection of serrated polyps. The materials included 94 clinical CT colonography cases with biopsy-confirmed serrated polyps. Our preliminary results indicate that both 3D DenseNets were able to detect serrated polyps at a high accuracy and high certainty. However, the 3D DenseNet with a larger number of input convolutions yielded more consistent certainties in the detection of different clinical pathologies of polyps than did the DenseNet with a smaller number of input convolutions. Our results indicate that the uncertainty quantification can provide constructive quantitative insights regarding the quality of the detections made by AI, and that serrated polyps can be detected automatically in CT colonography not only at a high detection accuracy but also at a high certainty.

Representation learning is one of the canonical objectives of most deep learning models. However, the learning of real-world clinical data is often compromised by their inherently imbalanced or long-tailed distribution wherein a few classes have significantly larger numbers of training instances than do the other classes. In this study, we investigated the representation learning of such long-tailed data distributions by the use of a deep mutual ensemble generative adversarial network. Our proposed framework consists of multiple powerful pre-trained discriminator networks that transfer knowledge to multiple individual untrained generator networks. During the training process, each generator learns to collaborate with the other generators. Additionally, each generator receives feedback from the individual discriminators in an adversarial manner. Especially, we explored the use of mutual information shared between the independent generators that makes our framework robust against misclassification of long-tailed data distributions in medical image analysis. We evaluated our proposed framework on four public datasets that represented different medical imaging modalities and imbalance ratios. Our experimental results show that our proposed framework benefits from ensemble learning and shared mutual learning, and achieves compelling results on several medical imaging benchmarks. Thus, our approach offers potential advantages over traditional deep learning in real-world applications.

Cardiovascular diseases are one of the strongest burdens in healthcare. If misdiagnosed, they can lead to life-threatening complications. This is especially true for aortic dissections, which may require immediate surgery depending on the categorization and still lead to late adverse events. Aortic dissection occurs when the aortic duct splits into two blood streams, the true and false lumina. The morphological characteristics of the aorta are therefore crucial for a clinician and provide vital support since they can be used to extract significant information for surgery and treatment planning. In this work, we revive a successful modeling technique – convolution surfaces – to model the lumina in aortic dissections. The skeleton of the lumina and local radial information are used to represent the true and the false lumen through convolution of local segments. Additionally, we introduce an optimization strategy based on a genetic algorithm to create the separation caused by the dissection flap.

In this work, we study convolutional neural network encoder-decoder architectures with pre-trained encoder weights for breast mass segmentation from digital screening mammograms. To automatically detect breast cancer, one fundamental task to achieve is the segmentation of the potential abnormal regions. Our objective was to find out whether encoder weights trained for breast cancer evaluation in comparison to those learned from natural images can yield a better model initialization, and furthermore improved segmentation results. We applied transfer learning and initialized the encoder, namely ResNet34 and ResNet22, with ImageNet weights and weights learned from breast cancer classification, respectively. A large clinically-realistic Finnish mammography screening dataset was utilized in model training and evaluation. Furthermore, an independent Portuguese INbreast dataset was utilized for further evaluation of the models. 5-fold cross-validation was applied for training. Soft Focal Tversky loss was used to calculate the model training time error. Dice score and Intersection over Union were used in quantifying the degree of similarity between the annotated and automatically produced segmentation masks. The best performing encoder-decoder with ResNet34 encoder tailed with U-Net decoder yielded Dice scores (mean±SD) of 0.7677±0.2134 for the Finnish dataset, and ResNet22 encoder tailed with U-Net decoder 0.8430±0.1091 for the INbreast dataset. No large differences in segmentation accuracy were found between the encoders initialized with weights pre-trained from breast cancer evaluation, and of those from natural image classification.