This study aims to characterize the effect of background tissue density and heterogeneity on the detection of irregular masses in breast tomosynthesis, while demonstrating the capability of the sophisticated tools that can be used in the design, implementation, and performance analysis of virtual clinical trials (VCTs). Twenty breast phantoms from the extended cardiac-torso (XCAT) family, generated based on dedicated breast computed tomography of human subjects, were used to extract a total of 2173 volumes of interest (VOIs) from simulated tomosynthesis images. Five different lesions, modeled after human subject tomosynthesis images, were embedded in the breasts and combined with the lesion absent condition yielded a total of VOIs. Effects of background tissue density and heterogeneity on the detection of the lesions were studied by implementing a composite hypothesis signal detection paradigm with location known exactly, lesion known exactly or statistically, and background known statistically. Using the area under the receiver operating characteristic curve, detection performance deteriorated as density was increased, yielding findings consistent with clinical studies. A human observer study was performed on a subset of the simulated tomosynthesis images, confirming the detection performance trends with respect to density and serving as a validation of the implemented detector. Performance of the implemented detector varied substantially across the 20 breasts. Furthermore, background tissue density and heterogeneity affected the log-likelihood ratio test statistic differently under lesion absent and lesion present conditions. Therefore, considering background tissue variability in tissue models can change the outcomes of a VCT and is hence of crucial importance. The XCAT breast phantoms have the potential to address this concern by offering realistic modeling of background tissue variability based on a wide range of human subjects, comprising various breast shapes, sizes, and densities.