CDRH/FDA, Division of Imaging and Applied Mathematics, Bldg. 62, Rm 3124, 10903 New Hampshire Avenue, Silver Spring, Maryland 20993-0002, United States
University of Iowa, Departments of Radiology and Biostatistics, 3170 Medical Laboratories, 200 Hawkins Drive, Iowa City, Iowa 52242-1077, United States
Comprehensive Access and Delivery Research and Evaluation Center, VA Health Care System, Iowa City, Iowa 52242-1077, United States
Modeling and simulation are often used to understand and investigate random quantities and estimators. In 1997, Roe and Metz introduced a simulation model to validate analysis methods for the popular endpoint in reader studies to evaluate medical imaging devices, the reader-averaged area under the receiver operating characteristic (ROC) curve. Here, we generalize the notation of the model to allow more flexibility in recognition that variances of ROC ratings depend on modality and truth state. We also derive and validate equations for computing population variances and covariances for reader-averaged empirical AUC estimates under the generalized model. The equations are one-dimensional integrals that can be calculated using standard numerical integration techniques. This work provides the theoretical foundation and validation for a Java application called iRoeMetz that can simulate multireader multicase ROC studies and numerically calculate the corresponding variances and covariances of the empirical AUC. The iRoeMetz application and source code can be found at the “iMRMC” project on the google code project hosting site. These results and the application can be used by investigators to investigate ROC endpoints, validate analysis methods, and plan future studies.
The area under the receiver operating characteristic (ROC) curve, denoted AUC, is a common endpoint in reader studies to evaluate medical imaging devices.1 ROC data from reader studies are confidence-of-disease ratings from clinicians (readers) evaluating images (cases). Therefore, the endpoint in a multireader multicase (MRMC) ROC study is affected by two important sources of variability—the readers and the cases—and in studies past the exploratory stage, we often want to account for both sources in our analyses.
Modeling and simulation are tools that help us to understand and investigate the distribution and statistical behavior of random quantities and estimators. Roe and Metz2 (R&M) proposed a simulation model that launched the study of different analysis methods and endpoints related to MRMC ROC studies. Their model was developed to validate the Dorfman, Berbaum, and Metz (DBM) method that compares AUCs from two modalities.3 The R&M model simulates ROC ratings according to a binormal model for each reader and generates data from a “fully crossed” study design where each patient is imaged by two or more modalities, with the resulting cases evaluated once by each reader.
With few modifications, the R&M model has been used to validate and characterize many other MRMC ROC analysis methods for almost two decades.4–10 The R&M model has also been used to investigate power and sizing methods for ROC studies,11,12 and adapted to yield discrete ROC ratings13,14 and explore alternative study designs.15 Finally, while AUC has been the primary reader performance measure analyzed with the R&M model to date, the R&M model has also been used to analyze binary performance measures16,17 and utility.18
The R&M model assumes a four-factor modality×reader×case×truth mixed-effects analysis of variance model for the ROC ratings. In the original work, reader and case effects and their interactions were assumed to be random and the variance components were chosen to be the same across truth states and across modalities. As we show below, the assumptions applied to the R&M model in the original work can be relaxed. In the current work, we generalize the notation to clarify that the variance components can depend on truth state and modality.
As discussed by Hillis,19 when the variance components are assumed to be the same across truth states and modalities, the R&M model has the following interpretations: (1) ROC ratings for each reader are generated from an equal-variance binormal model (i.e, a binormal model such that variances of the nondiseased and diseased ROC ratings are equal); and (2) the expected differences (or separations) between the nondiseased and diseased ROC ratings vary across readers, with the separations having the same variance for each modality. This last result implies that for a simulation study that assumes equal AUCs across modalities (i.e., a null-hypothesis study), the resulting AUC estimates will have the same variance for each modality. It is natural to question this assumption, especially when comparing an imaging modality with and without a computer aid. Beiden et al.20 found that the reader variability of readers’ AUCs was much smaller when using a computer aid in classifying microcalcifications in mammograms compared to those without the aid.
When fitting an ROC curve with a binormal model, it is generally recognized that for real data the distributions of the latent decision variables of the diseased and nondiseased ROC ratings will often have different variances, with the diseased distribution typically wider. This causes the ROC curve to be unsymmetric about the negative diagonal. Such unsymmetric ROC curves have been seen as far back as the early psychophysical experiments of the 1960s21,22 and in recent studies evaluating medical imaging modalities.19,23–26 Unsymmetric ROC curves have motivated other models of ROC ratings27–29 and are sometimes characterized by a mean-to-sigma ratio defined as the difference of the binormal means divided by the difference of the binormal standard deviations across truth states in an unequal-variance binormal model.19,21,26 For this reason, Hillis19 introduced an unequal-variance binormal model by allowing some of the variance components to depend on truth but with some additional constraints.
In this paper, our main purpose is to present the exact nature of the relationship between the R&M model inputs (these include the model parameters and numbers of readers and cases) and the means, variances and covariances of the resulting reader-averaged empirical AUC estimates. R&M note that they were not able to determine such a relationship. Knowledge of this relationship builds on earlier work by Gallas et al.16 and allows an investigator to, among other things, (1) verify that the simulation model has been correctly programmed by comparing parameter estimates based on the simulations to the true values of the parameters; and (2) quantify the bias of AUC variance estimates by similarly comparing simulation results to the true values. We consider this paper to be an important first step toward our ultimate aim of being able to calibrate a simulation model that will produce data that matches a real data set with respect to the estimated parameters from an analysis method, such as that proposed by Obuchowski and Rockette.30
In this paper, we begin by generalizing the notation of the R&M model by allowing all of its variance components to depend on both modality and truth state. Then, we present and validate equations for computing the population variances and covariances for empirical AUC estimates computed from data simulated from the generalized R&M model. The generalized model includes the original R&M model (with the equal variance-components assumption) and the unequal-variance model proposed by Hillis19 as special cases. Although these two special cases are sufficient for many situations, we anticipate that researchers may want to use other special cases of the generalized R&M model, or the generalized model itself for simulating data. Presenting equations for the generalized model eliminates the need to derive equations for each special case in the future.
The R&M simulation model is for simulating rating data that emulate an ROC reader-performance study that has N0 nondiseased cases and N1 diseased cases that are interpreted and rated by NR readers. Here, we focus on a fully crossed study design to compare two modalities, denoted by A and B, where “fully crossed” refers to the data collection: all the readers rate all the cases in both modalities with respect to confidence of disease. The result of such a study is a dataset with 2×(N0+N1)×NR ROC ratings [(modalities)×(cases)×(readers)]. R&M denote the ROC ratings by Xijkt, where i denotes modality (i= “A” or “B”), j denotes reader, k denotes case, and t denotes truth (t=0,1≡nondiseased, diseased).
Using the notation of R&M, the model is given by Display Formula
The random terms in the R&M model are all independent zero-mean Gaussian random variables: R is the reader effect, C is the case effect, [RC] is the reader×case effect, [τR] is the modality×reader effect, [τC] is the modality×case effect, [τRC] is the modality×reader×case effect, and E is an independent random error term. The corresponding variance components are denoted σR2,σC2,σRC2,στR2,στC2,στRC2, and σE2. The independent error term Eijkt can be attributed to a reader’s inability to exactly reproduce their ROC rating for a case; E is sometimes referred to as internal noise or reader jitter. R&M pointed out that the pure error and the three-way interaction variance components cannot be separately estimated “without multiple readings of each case in each modality by all readers.” As such, they defined an aggregate term εijkt=[τRC]ijkt+Eijkt.
Here, we generalize the R&M model, clarifying that the model is, in fact, a four-factor model. In addition to modality, reader, and case, truth is a factor. In particular, truth has a hierarchical relationship with cases:31 each case can have only one truth state and cases are “nested” within truth. Mathematically, we will use α to express the truth factor and rewrite the R&M model as Display Formula
Now, the four-factor structure of the model is clearly shown in Eq. (2). We point out that this model includes only interactions with truth or effects nested within truth; in particular, there are no effects for modality alone or reader alone. The rationale for omitting the modality and reader effects is that these terms would have no effect on the ROC curve for a given reader and test, since the ROC curve is invariant to location shifts of the decision variable. We note that the fixed interaction [τα]it allows different modalities to have different ROC curves.
Given the new notation for the R&M model, we now generalize it to allow the variance components to depend on modality and truth.
There are three random-effect terms that do not include modality: [Rα]jt,[Cα]kt,[RCα]jkt. They correspond to six variance components: σR02,σC02,σRC02 are for nondiseased cases (t=0) and σR12,σC12,σRC12 are for diseased cases (t=1). The sum of these six variance components that do not depend on modality is Display Formula
There are three random-effect terms that include modality: [τRα]ijt, [τCα]ikt, and [τRCα]ijkt. They correspond to 12 variance components: σAR02,σAC02,σARC02 are for modality A, nondiseased cases (i=A;t=0), σAR12,σAC12,σARC12 are for modality A, diseased cases (i=A;t=1), σBR02,σBC02,σBRC02 are for modality B, nondiseased cases (i=B;t=0), and σBR12,σBC12,σBRC12 are for modality B, diseased cases (i=B;t=1). The sums of the variance components that are specific to modality A and B are Display Formula
The original R&M model is a special case of the generalized model. All we need to do is assume, as R&M did, that the variance components of the ROC ratings do not depend on modality or truth. We can replicate the original R&M model by setting generalized R&M model variance components equal to the original ones as given in Table 1. Other simplifications can be similarly handled.
Here, we examine the expected value of the empirical estimate of AUC, also known as the trapezoidal estimate,32 given the R&M model. Without loss of generality, we focus on modality A. A similar discussion can be derived for modality B.
The estimated reader-averaged AUC for modality A is Display Formula
As we consider the expected reader-averaged AUC for modality A, we are averaging over readers and cases as we average over the ROC ratings. We express the expectation of AUC^A [Eq. (6)] as Display Formula
In Eq. (6), XAjk′1−XAjk0 is nothing more than the difference of a few fixed effects and many zero-mean random effects. Specifically, it is a normal random variable with a mean and variance given by Display Formula
Here, we turn our attention to variances and covariances. Both appear in the variance of the difference of estimated reader-averaged AUCs: Display Formula
Without loss of generality, we first discuss the variance of AUC^A. A similar discussion can be given for the variance AUC^B. We will then discuss the covariance of AUC^A and AUC^B.
The variance of a single modality can be decomposed into different representations. We shall use the success-moment representation that can be derived using U-statistics;33 i.e., Display Formula
The particular moment will be clear from the subscripts as, from this point forward, we will not allow different subscripts to take on the same value. The interpretation of each moment is driven by the unique subscripts that appear. For example, in the expression [M̲A]1=E(sAjkk′sAjkk′) we see that the subscripts are identical on both success observations; therefore, the expression is the expected value for a randomly selected reader reading a randomly selected diseased and nondiseased case. In contrast, in the expression [M̲A]8=E(sAjkk′sAj′k′′k′′′) we see that every subscript on the first success observation is different from its counterpart on the second success observation; therefore, the expression is an expected value over a pair of randomly selected readers (that are unique), reading randomly selected diseased and nondiseased cases (that are all unique).
To derive the eight moments, we first treat two special cases, then we discuss the rest. The two special cases are the examples above: [M̲A]1 and [M̲A]8. In the first special case, the success observations are identical and both are equal to one or zero (ROC ratings from the generalized R&M model are continuous); consequently, their product equals one or zero. Therefore, [M̲A]1=E(sAjkk′)=AUCA. In the second special case, the success observations are independent because they come from different readers reading different nondiseased and diseased cases. Therefore, the expected value can be factored and [M̲A]8=E(sAjkk′)E(sAj′k′′k′′′)=AUCA2.
The remaining moments of M̲A (moments 2 to 7) can be written as Display Formula
Given a fully-specified R&M model, we can compute the moments above (and the moments corresponding to the variance of AUC^B) using basic numerical integration methods. The derivation of the expression above follows a common outline. We exemplify the derivation for one of the moments in Appendix 1. Here is the outline:
The covariance of AUC^A and AUC^B can be decomposed into a success-moment representation analogous to the variance; namely, Display Formula
There is only one special case this time: [M̲AB]8. The success observations for this moment are again independent because they again come from different readers reading different cases. Therefore, the expected value can be factored and [M̲AB]8=E(sAjkk′)E(sBj′k′′k′′′)=AUCAAUCB.
The remaining moments of M̲AB (moments 1 to 7) can be written as Display Formula
Given a fully specified R&M model, we can compute the moments above using basic numerical integration methods. The derivation of the expression above follows the same common outline as above. We illustrate the derivation for [M̲AB]5 in Appendix 1.
In this paper, we mimic the Monte Carlo (MC) simulation experiments run by R&M; however, we modify their experiments by perturbing the input variance components so that the variance components of the ROC ratings depend on modality and truth state. The purpose of the experiments here is to validate the numerical calculations given above against simulated results. Like R&M, we simulate data sets that are fully crossed: all readers read all images in both modalities with no re-reading. In what follows, we describe the original R&M simulations and detail how the simulations in this paper are built on the original.
R&M set to zero the effect for nondiseased cases and all four modality-truth interaction effects: Display Formula
Consequently, the expected differences in ROC ratings for both modalities are equal to the effect for diseased cases (ΔA=ΔB=α1). R&M considered three levels of this effect; namely, α1=0.75, 1.50, and 2.50. These three levels ultimately control the expected AUCs via Eqs. (8) and (9). Now, since the variances of the random effects did not depend on modality, they effectively simulated a null-hypothesis experiment: AUCA=AUCB. The experiments in this paper will use the same values for the fixed effects. However, because we perturb some variance components in order to generate ROC ratings that depend on modality and truth state (described below), AUCA will not equal AUCB and they will both be different from the AUCs of the original R&M simulation.
R&M investigated experiments with three or five readers and case-sets that ranged from 50 to 400. They also explored ratios of case mixes (nondiseased to diseased) of 1:1 and 9:1. We shall only investigate experiments with five readers, 50 nondiseased cases, and 50 diseased cases. We feel that limiting the investigations related to the size of the experiment is appropriate since the purpose here is more modest than the original purpose (validating a hypothesis test).
R&M based their original simulation experiments on correlation estimates found in actual ROC analyses using the CORROC algorithm,34 which assumes an underlying bivariate binormal distribution. They investigated high versus low data correlation between and within readers (due to reading the same cases) at both high and low values of total reader variability. We explore a similar set of variance structures, but we perturb them to get variance structures that are different across modality and truth state. Specifically, we halve the variances of effects that involve modality A and truth=diseased, and we double the variances of effects that involve modality B and truth=diseased: Display Formula
This perturbation causes σA2 to be different from σB2, which then causes AUCA to be different from AUCB [Eq. (9)].
In total, we explore two levels of between- and within-reader data correlation, four levels of reader variability, and three levels of performance in a factorial fashion for a total of 24 simulation configurations. Details are provided in Appendix 2 that will allow the interested reader to replicate our experiments.
Figure 1 shows the variances and covariances of AUC^A and AUC^B (numerical results versus estimates) of the 24 simulation experiments described above: 8 configurations×3 performance levels. The x-axis shows the variances by numerical integration. The y-axis shows the MC estimates of variance given 1000 MC trials. There is some variability along the line of equality due to the relatively small number of MC trials. The absolute value of the relative differences between the numerical results and the estimates averaged over all 24 simulation configurations is 5.2%. For 100,000 MC trials, the averaged absolute value of the relative differences is 0.5% and the variability along the line of equality is not visible (not shown).
The variances and covariances (estimates versus numerical results) of the 24 generalized R&M simulation configurations described in the text. The estimates are described in the text. The results here are for 1000 Monte Carlo (MC) trials. Differences converge as the number of MC trials increases.
In this paper, we generalized the R&M model by allowing all of its variance components to depend on both modality and truth state. This will allow investigators to model and simulate MRMC ROC studies that better fit their data. Additionally, we presented and validated equations for computing the population variances and covariances for empirical AUC estimates computed from data simulated from the generalized R&M model. These equations show the core relationships between the ROC data and the reader-averaged AUCs. These equations and relationships should help investigators to validate new MRMC variance and covariance estimation methods, explore novel study designs, size future trials, and model MRMC ROC data.
For the interested investigator, the first author has made available a stand-alone Java application called iRoeMetz. iRoeMetz and its source code can be found in 35. iRoeMetz can simulate ROC studies according to the generalized R&M model and numerically calculate the expected moments and variances found in this paper. Additionally, all the original R&M simulation configurations and those used in this paper can be downloaded and tested.
Future work is needed in this area, investigating real data sets and calibrating the different possible R&M models of decision scores to be consistent with real data estimates of AUC variance components. To cover the full range, datasets that study different anatomical locations and diseases, and different imaging modalities (with and without computer aids as appropriate) are needed. The results of such future work would provide investigators with more information and examples to help them calibrate the R&M model (or any other model) to their situation so that they can appropriately size future reader studies. We are collecting datasets for this work for public sharing toward this goal; we welcome any and all contributions.
There are other interesting future efforts that could support the need mentioned above. One direction of study could link the success moments at the core of the work here to the parameters of the Obuchowski and Rockette30 model for AUC observations. Another direction of study could be to uncover the inverse mapping: given the parameters of the Obuchowski and Rockette model for AUC observations, solve for the variance components of the ROC ratings that are consistent with them. It is not currently known whether the solution is unique or if it can be treated efficiently and effectively.
In this appendix, we derive one of the product moments, [M̲AB]5, that enters into the computation of the covariance between AUC^A and AUC^B via Eq. (13). Adapting the results to the other moments that are required for computing the variance of AUC^A and the variance of AUC^B is straightforward.
Although the generalized R&M model has many parameters and the variances and the covariance of the reader-averaged AUCs are fairly complex, all of the success moments are derived the same way and the results have a common form. The derivations follow a common outline. The results are straightforward 1-D integrals involving the normal distribution that can be calculated using basic numerical integration methods. We illustrate the derivation for [M̲AB]5.
Here is the derivation outline:
Step 1: The fifth product moment is Display Formula
Step 2: Here, we first write the moment as the expected value of a conditional expected value, where the random effects found in Step 1 are fixed; namely, Display Formula
Step 3: In the expected value concerning modality A, the argument of the success function is a normal random variable Z with mean and conditional variance given by Display Formula
Step 4: Last, we see that the only randomness that remains above is the difference [Cα]k′1−[Cα]k0. This difference is a normal random variable Y=[Cα]k′1−[Cα]k0 with mean zero and variance σΩ2−σΩ(5)2=σC02+σC12. After a change of variables, X=Y/σΩ2−σΩ(5)2 we can effectively integrate over the random variables that were initially fixed, [Cα]k′1,[Cα]k0, with the following: Display Formula
Here, we document the simulation configurations used in this paper. We start by defining original R&M variance components. We then map them to generalized R&M variance components (Table 1). Finally, we perturb them to get variance structures that are different across modality and truth state.
The simulation configurations in this paper explore the following settings in a factorial fashion for 5 readers, 50 nondiseased cases, and 50 diseased cases:
Let us consider the simulation with the high between- and within-reader data correlation and the lowest reader variability. After mapping and perturbing, the generalized R&M variance components are Display Formula
This variance structure used with the lowest level of performance (ΔA=ΔB=α1=0.75) leads to the following expectations of performance and uncertainty for 5 readers, 50 nondiseased cases, and 50 diseased cases: Display Formula
This research was partially supported by the National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health under Award Number R01EB013667. The content is solely the responsibility of the authors and does not necessarily represent the offical views of the National Institutes of Health, the Department of Veterans Affairs, or the United States government.
Brandon D. Gallas provides mathematical, statistical, and modeling expertise to the evaluation of medical imaging devices at the FDA. His main areas of contribution are in the design and statistical analysis of reader studies, image quality, computer-aided diagnosis, and imaging physics. Before working at the FDA, he was in Dr. Harrison Barrett’s research group at the University of Arizona, earning his PhD from the Graduate Interdisciplinary Program in Applied Mathematics.
Stephen L. Hillis is a research professor in the Departments of Radiology and Biostatistics at the University of Iowa, and Senior Statistician for the Iowa City VA Healthcare System. He earned his PhD in statistics from the University of Iowa in 1987 and is the author of 85 peer-reviewed articles. Since 1998 his research has focused on methodology for analyzing diagnostic radiological imaging studies.
Brandon D. Gallas and Stephen L. Hillis"Generalized Roe and Metz receiver operating characteristic model: analytic link between simulated decision scores and empirical AUC variances and covariances", J. Med. Imag. 1(3), 031006 (Sep 25, 2014). ; http://dx.doi.org/10.1117/1.JMI.1.3.031006
Some tools below are only available to our subscribers or users with an online account.
Download citation file:
Customize your page view by dragging & repositioning the boxes below.
Shibboleth is an access management service that provides single sign-on to protected resources. It replaces the multiple user names and passwords necessary to access subscription-based content with a single user name and password that can be entered once per session. It operates independently of a user's location or IP address. If your institution uses Shibboleth authentication, please contact your site administrator to receive your user name and password.
Access to SPIE eBooks is limited to subscribing institutions and is not available
as part of a personal subscription. Print or electronic versions of individual SPIE
books may be purchased via SPIE.org.
Shibboleth is access management services that provide single sign-on to protected resources. It replace the multiple user names and passwords necessary to access subscription-based content with a single user name and password that can be entered once per session. It operates independently of a user's location or IP address. If your institution uses Shibboleth authentication, please contact your site administrator to receive your user name and password.