1.

Introduction

Estimating volumes and masses of total body components is important for research of cancer, joint replacement, and exercise physiology.¹ Full body CT scans can be used to calculate whole body compositions directly. However, it is hard to acquire a typical full body CT in the usual medical context due to the intense radiology dose. Mourtzakis et al.² proposed that body components measured on abdomen or thigh slices are highly correlated with the mass of whole-body tissues. Thus, accurate segmentation of thigh slices can quantify tissue area properties to estimate body composition without requiring additional irradiation or examinations. Thus, this paper aims to segment muscle, fat, and bones from 2D thigh and lower leg CT slices.

Several recent techniques have been proposed to address thigh and lower leg segmentation on CT images. Senseney et al.³ proposed an automatic region growing method using a morphology operation and threshold to extract bone muscle and fat in CT thigh and abdomen images. Tan et al.⁴ proposed using a variational Bayesian Gaussian mixture model to cluster fat, marrow, muscle bone, and air on three-dimensional (3D) CT scans. Felinto et al.⁵ proposed using a Gaussian mixture model and relative position to cluster similar tissues for intermuscular fat and muscles segmentation. With impressive performance of the deep neural network-based segmentation, Zhu et al.⁶ applied the H-DenseU-Net on MRI lower leg data of children with and without cerebral palsy. Rohm et al.⁷ created a 3D heterogeneous MRI lower leg dataset and trained a convolution network to segment muscle.

Deep learning methods show impressive performance in segmentation tasks. However, this performance depends on sufficient human annotation.⁸ In the medical imaging field, human annotation requires professional knowledge, which is very time-consuming and thus expensive. To avoid annotating new data, many researchers used common data augmentation methods such as rotation, intensity shift, and scaling to artificially enhance the diversity and quality of the training data.⁹ Image synthesis is another data augmentation method. Generative adversarial networks (GANs)¹⁰ have been utilized to synthesize new labeled data for segmentation. However, GAN is notorious for training and is hard to implement in practical tasks.¹¹ The main limitation of data augmentation is data bias generated during data augmentation process. To preserve original data distribution, leveraging the power of unannotated data is another solution to train a model with limited annotation data. Chen et al. proposed using self-supervised learning with image context restoration to achieve brain tumor segmentation with a limited dataset.¹² Instead of self-supervised learning, transfer learning is another way to train with limited label data. First, a model is trained from scratch on a large-scale dataset with a similar task. Then, the model is fine-tuned with human annotated data. Tajbakhsh et al.¹³ showed that a fine-tuned network could outperform networks that were trained from scratch with better robustness.

To better segment muscle, cortical bone, internal bone, subcutaneous fat, and intermuscular fat with limited annotated data, we propose a two-stage transfer learning-based framework. We use an approximate handcrafted method to generate pseudo labels for 1883 thighs to train the model in the first stage and fine-tune it with 125 human label thighs in the second stage to achieve segmentation. We test the model on the thigh slice and use the lower leg slice as external data to demonstrate the generalizability of the proposed framework. The target tissue and corresponding legend are shown in Fig. 1. This paper is the significant extension of our prior accepted work¹⁴ of SPIE 2022.

Fig. 1

The first row and second row of (a) represent the middle thigh and lower leg from the same subject, respectively. The left column is the original CT image, and the right column is the target tissues label. Each tissue has a different area, and the imbalance of area makes the segmentation of sparse tissue (intermuscular fat) challenging. The area of each tissue is shown in (b).

2.

Material and Methods

We designed a two-stage coarse-to-fine deep learning method to achieve thigh and lower leg segmentation on low-dose CT slices with deep learning. We first split a CT thigh slice into single left and right thigh images. In the first stage, we train the deep neural network with approximate handcrafted labels. In the second stage, we fine-tune the model from the first stage with human expert labels to recover more details.

2.2.

Create Pseudolabel for Thigh

Each CT slice has specific intensity units for each tissue. We use a CT window of [$-190,-30$] HU for fat, [30,80] HU for muscle, and [1000,∞] HU for bones.¹⁵ We propose the following pipeline with seven steps to extract five target tissues coarsely using CT intensity and morphology.

• (1) Create a cortical bone binary mask image with a threshold of 1000.

• (2) Invert the cortical bone mask and find the connected region with an area that is smaller than the half image ($256\times 256/2$) as the internal bone mask.

• (3) Use a threshold of 0 HU to binarize the thigh image and create a muscle mask.

• (4) Fill the holes for result (3).

• (5) Subtract the muscle mask from step (4) to create an intermuscular fat mask based on the assumption that intermuscular fat is within the muscle.

• (6) Binarize the thigh image with a threshold of $-500$ HU.

• (7) Subtract the result of step (4) from step (6) to create a subcutaneous fat mask. Five coarse approximate segmentation masks are shownin Fig. 2. They are fused into one mask before fed into the deep neural network.

Fig. 2

The proposed hierarchical coarse-to-fine thigh segmentation including three parts: (1) Use the threshold and morphology to generate coarsely segmented pseudo labels. (2) Feed pseudo labels into deep learning model, and train the model from scratch. (3) Use the optimized model from previous stage as initialization, and fine-tune the model with limited expert labels. The model from the first and second stages is optimized separately.

3.

Results

Figure 3 compares the DSC of the muscle, cortical bone, inner bone, subcutaneous fat, and intermuscular fat between U-Net++ and U-Net using only human labels, in stage 1 and stage 2. The boxplots presented are evaluated across 73 single thighs. Table 2 shows the mean DSC of each tissue for all six methods. Overall, the average DSC across all five tissues of U-Net++ in the second stage is significantly better than the other five methods. Except for subcutaneous fat, the proposed method has the highest mean DSC of the remaining tissues. The proposed method makes the largest improvement from 0.681 to 0.782 on mean DSC for sparse and small intermuscular fat compared with U-Net trained only with human labels. Figure 4 compares the qualitative result produced by all six methods. Compared with U-Net in stage 2, the proposed method yields superior performance and segments more details of intermuscular fat.

Fig. 3

The DSC comparison of thigh images using U-Net trained only with human labels, U-Net++ trained only with human label, U-Net in stage 1, U-Net++ in stage 1, U-Net in stage 2, and U-Net++ in stage 2 in boxplots of five target tissues.

Table 2

The mean DSC for each tissue of each method for the thigh CT image. The highest result is bolded.

Fig. 4

Qualitative representation of the thigh slice segmentation. (a) Three randomly selected source CT images after applying window [$-150,100$]. (b) The segmentation from U-Net only trained with human labels. (c) The segmentation from U-Net++ only trained with human labels. (d) and (e) The segmentation using network U-Net and U-Net++ in stage 1, respectively. (f) and (g) The segmentation by network U-Net and U-Net++ in stage 2, respectively. (h) The ground truth. The yellow arrow points to the large difference across the methods and ground truth. The DSC values only show intermuscular fat segmentation performance for reference.

To test the generalizability of the proposed methods, we applied the model on preprocessed lower leg images. The experimental setting is the same as in the thigh experiment. Figure 5 compares the performance of all six methods on lower leg images. Table 3 shows the mean DSC of each tissue of all six methods on lower leg images. The average DSC of all tissues of the proposed method decreased from 0.927 to 0.823 when compared with the thigh experiment, but significantly outperformed all other methods except U-Net in the second stage. The U-Net trained only with human labels has the highest mean DSC 0.922 in cortical bones, and the U-Net++ trained only with human labels has the highest mean DSC 0.9 in internal bones. Figure 6 shows the confidence level of the bone result with qualitative representations.

Fig. 5

The DSC comparison of the lower leg image among using U-Net trained only with human labels, U-Net++ trained only with human labels, U-Net in stage 1, U-Net++ in stage 1, U-Net in stage 2, and U-Net++ in stage 2 of five target tissues.

Table 3

The mean DSC for each tissue of each method for the lower leg CT image. The highest result is bolded.

Fig. 6

Qualitative representation of the lower leg slice segmentation. (a) The source CT image after applying window [$-150,100$]. (b) The segmentation from U-Net only trained with human labels. (c) The segmentation from U-Net++ only trained with human labels. (d) and (e) The segmentation using network U-Net and U-Net++ in stage 1, respectively. (f) and (g) The segmentation by network U-Net and U-Net++ in stage 2, respectively. (h) The ground truth. The text below each image is the internal bone DSC.

After training on pseudo labels, we want to investigate the relationship between the number of human expert data and performance of the model in the second (fine-tune) stage. We fed 1, 5, 10, 20, 30, 60, 90 and all expert human label thighs to fine-tune the model from the first stage. The fine-tuning process is repeated 10 times. Each time the data are randomly picked from training cohort (125 thighs) and inference on the test cohort (73 thighs). The distribution of the mean dice of each retraining is shown in Fig. 7. When only feeding one thigh to the model, the variance of the mean DSC is largest compared with the others. Also, the variance becomes smaller when increasing the feeding data. When feeding 30 thighs, the mean DSC of 10 retraining is 0.924, almost equivalent to the mean DSC 0.931 of feeding all training data.

Fig. 7

The relationship between the mean DSC and added data for each fine-tuning. The violin plot includes 10 data points. Each data point represents the mean DSC across all tissues of the test cohort in one fine-tuning process.

We apply the proposed models on additional CT scans of thighs and lower legs, which do not have human generated label maps for comparison. We overlay the segmentation results on CT images with a colormap of the human review of each segmentation result to find outliers. Ten out of 3982 thigh images and 136 out of 17,878 lower leg images fail human review and are regarded as outliers. Figure 8 shows four outliers from thigh and lower leg images, respectively.

Fig. 8

The outliers from the thigh and lower leg. The first row and third row are segmentation result on the thigh and lower leg, respectively. The second and fourth row are the CT image after applying the window. Each column represents an outlier from the thigh and lower leg, respectively.

4.

Conclusion and Discussion

Herein, we proposed a transfer learning-based method to achieve accurate and robust thigh tissue segmentation, focusing on muscle, cortical bone, internal bone, subcutaneous fat, and intermuscular fat. The proposed framework achieved accurate segmentation on thigh CT slices with limited human labels. Compared with the method only trained with human expert labels, the superior performance of the proposed framework demonstrates the effectiveness of two-stage training. We applied the model only trained on the thigh slice to the lower leg image. The results show that our model still recognizes muscle and fat with high agreement, which demonstrates that the proposed framework has the generalizability to similar anatomical structures of CT images. Then, we analyzed the relationship between the size of the fine-tuning dataset and performance. The result shows that the proposed framework can use a smaller size of training cohort to keep the same performance as all training data, which indicates that the proposed framework has the potential to fully exploit the data and increase data efficiency.

In the medical image segmentation, label annotation is always a challenging problem for the deep learning methods. Similar to Ref. 18, the proposed work leverages the power of cheaper-to-acquire pseudo labels to achieve high-performance segmentation. Both methods work on the assumption that deep learning models can explore general patterns from pseudo labels and inference on unseen data. The primary difference between the approaches is that the proposed method used limited human labels to fine-tune model to improve performance of tissue with an appearance that is hard for pseudo labels to capture.

The proposed method can be extended to 3D CT images with target tissue that is strongly related to intensity. The automatic method based on intensity can create pseudo labels for target tissue. Different from 2D slices, 3D volumes require more data to train more parameters in the deep neural network. We expect that the number of pseudo labels and human provided labels needs to be increased for successful application of the method.

One major limitation of the proposed method is that the performance of bone structure is inferior to models only trained with human labels. As shown in Fig. 7, the proposed method might regard subcutaneous fat around cortical bone as internal bone, which leads to inferior performance. Except for the domain shift between the thigh and lower leg, the reason could be that the boundary of the pseudo label between cortical bone and internal bone is not clearly defined, which made the model misclassify subcutaneous fat around cortical bones. Thus, how to improve quality of the pseudo label is an important future research direction.

The training dataset is all mid-thigh images. The model is adapted to the anatomy of mid-thigh and applies well on lower leg images because of the similar anatomy. The CT scans including the femoral head have a different anatomy, bringing varied CT unit scales, particularly in the bone part. The model may segment muscle, intermuscular fat, and subcutaneous fat. However, it cannot recognize targeted tissue from the femoral head correctly. Thus, how to develop a model that can handle different positions of the lower limb will be an important future research direction.

Normal training images have a fixed anatomy pattern and classes of target tissues. The pseudo labels are created with the assumption that there is no abnormal bone intensity or shape. When applying the model on abnormal bones, the method is biased to recognize the bone pathology as the normal bone tissues and segment them into incorrect labels. In the future, the pseudo labels method may be improved to segment the abnormal bone part and feed more abnormal cases into the model, so the model may recognize the pathology part.

In summary, the proposed pipeline has potential to be applied on other medical scenes with low human effort, which makes better use of human expert labels.