Effects of the continuous and discrete confidence rating scales in ROC observer studies

Lubomir M. Hadjiiski; Heang-Ping Chan; Berkman Sahiner; Mark A. Helvie M.D.; Marilyn A. Roubidoux

doi:10.1117/12.595725

6 April 2005 Effects of the continuous and discrete confidence rating scales in ROC observer studies

Lubomir M. Hadjiiski, Heang-Ping Chan, Berkman Sahiner, Mark A. Helvie M.D., Marilyn A. Roubidoux

Proceedings Volume 5749, Medical Imaging 2005: Image Perception, Observer Performance, and Technology Assessment; (2005) https://doi.org/10.1117/12.595725
Event: Medical Imaging, 2005, San Diego, California, United States

Abstract

We previously conducted an observer study evaluating radiologists' performance for characterization of mammographic masses on serial mammograms with and without CAD. 253 temporal image pairs (138 malignant and 115 benign) from 96 patients containing masses on serial mammograms were used. The interval change characteristics of the masses on each temporal pair were analyzed by our CAD program to differentiate malignant and benign masses. The classifier achieved a test Az value of 0.87 for the data set. Eight MQSA radiologists and 2 fellows assessed the temporal masses and provided estimates of the likelihood of malignancy (LM) and BI-RADS assessment without and then with CAD. The LM estimates were provided on a quasi-continuous confidence-rating scale (CRS) of 1 to 100. In the current study we investigated the effects of using discrete CRS with fewer categories on ROC analysis. We simulated three discrete CRSs containing 5, 10, and 20 categories by binning the radiologists’ LM quasi-continuous ratings. For the ten radiologists, without CAD, the average Az in estimating the LM for the 5, 10, 20 and 100 category CRSs were 0.788, 0.786, 0.785, and 0.787, respectively. With CAD, the observers' Az improved to 0.845, 0.843, 0.844, and 0.843, respectively. The improvement was statistically significant (p<0.011) for each CRS. The partial area index for the four CRSs without CAD was 0.198, 0.204, 0.200, and 0.206, respectively. With CAD the partial area index was also significantly improved to 0.369, 0.365, 0.369, and 0.366, respectively (p<0.006 for all CRSs). The use of continuous and discrete confidence-rating scales in this study had minimal effect on the analysis of observer performance.

Citation Download Citation

Lubomir M. Hadjiiski, Heang-Ping Chan, Berkman Sahiner, Mark A. Helvie M.D., and Marilyn A. Roubidoux "Effects of the continuous and discrete confidence rating scales in ROC observer studies", Proc. SPIE 5749, Medical Imaging 2005: Image Perception, Observer Performance, and Technology Assessment, (6 April 2005); https://doi.org/10.1117/12.595725

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available

Members: $17.00

Non-members: $21.00 ADD TO CART

PROCEEDINGS
7 PAGES

DOWNLOAD PAPER SAVE TO MY LIBRARY

GET CITATION

RIGHTS & PERMISSIONS

Get copyright permission Get copyright permission on Copyright Marketplace

KEYWORDS

Computer aided diagnosis and therapy

Mammography

Computer aided design

Statistical analysis

Chromium

Biopsy

Analytical research

Show All Keywords

Keywords/Phrases

Search In:

Publication Years