Preface xiv
Acknowledgments xvi
Part I Basic Concepts and Methods 1
1 Introduction 3
1.1 Diagnostic Test Accuracy Studies 3
1.2 Case Studies 5
1.2.1 Case Study 1: Parathyroid Disease 5
1.2.2 Case Study 2: Colon Cancer Detection 6
1.2.3 Case Study 3: Carotid Artery Stenosis 7
1.3 Software 8
1.4 Topics Not Covered in This Book 8
2 Measures of Diagnostic Accuracy 9
2.1 Sensitivity and Specificity 9
2.1.1 Basic Measures of Test Accuracy: Case Study 2 11
2.1.2 Diagnostic Tests with Continuous Results: The Artificial Heart Valve Example 12
2.1.3 Diagnostic Tests with Ordinal Results: Case Study 1 13
2.1.4 Effect of Prevalence and Spectrum of Disease 14
2.1.5 Analogy to ¿ and ß Statistical Errors 14
2.2 Combined Measures of Sensitivity and Specificity 15
2.2.1 Problems Comparing Two or More Tests: Case Study 1 15
2.2.2 Probability of a Correct Test Result 15
2.2.3 Odds Ratio and Youden's Index 16
2.3 ROC Curve 17
2.3.1 ROC Curves: Artificial Heart Valve and Case Study 1 17
2.3.2 ROC Curve Assumption 18
2.3.3 Smooth, Fitted ROC Curves 19
2.3.4 Advantages of ROC Curves 19
2.4 Area Under the ROC Curve 20
2.4.1 Interpretation of the Area Under the ROC Curve 20
2.4.2 Magnitudes of the Area Under the ROC Curve 21
2.4.3 Area Under the ROC Curve: Case Study 1 21
2.4.4 Misinterpretations of the Area Under the ROC Curve 23
2.5 Sensitivity at Fixed FPR 25
2.6 Partial Area Under the ROC Curve 25
2.7 Likelihood Ratios 26
2.7.1 Three Examples to Illustrate Likelihood Ratios 27
2.7.2 Limitations of Likelihood Ratios 28
2.7.3 Proper and Improper ROC Curves 29
2.8 ROC Analysis When the True Diagnosis Is Not Binary 30
2.9 C-statistics and Other Measures to Compare Prediction Models 32
2.10 Detection and Localization of Multiple Lesions 33
2.11 Positive and Negative Predictive Values, Bayes' Theorem, and Case Study 2 35
2.11.1 Bayes' Theorem 36
2.12 Optimal Decision Threshold on the ROC Curve 38
2.12.1 Optimal Thresholds for Maximizing Classification 38
2.12.2 Optimal Threshold for Minimizing Cost 39
2.12.3 Optimal Decision Threshold: Rapid Eye Movement as a Marker for Depression Example 39
2.13 Interpreting the Results of Multiple Tests 40
2.13.1 Parallel Testing 40
2.13.2 Serial, or Sequential, Testing 41
3 Design of Diagnostic Accuracy Studies 45
3.1 Establish the Objective of the Study 45
3.2 Identify the Target Patient Population 49
3.3 Select a Sampling Plan for Patients 50
3.3.1 Phase I: Exploratory Studies 50
3.3.2 Phase II: Challenge Studies 50
3.3.3 Phase III: Clinical Studies 52
3.4 Select the Gold Standard 56
3.5 Choose a Measure of Accuracy 61
3.6 Identify Target Reader Population 63
3.7 Select Sampling Plan for Readers 64
3.8 Plan Data Collection 64
3.8.1 Format for Test Results 64
3.8.2 Data Collection for Reader Studies 65
3.8.3 Reader Training 71
3.9 Plan Data Analyses 72
3.9.1 Statistical Hypotheses 72
3.9.2 Planning for Covariate Adjustment 73
3.9.3 Reporting Test Results 75
3.10 Determine Sample Size 77
4 Estimation and Hypothesis Testing in a Single Sample 79
4.1 Binary-scale Data 80
4.1.1 Sensitivity and Specificity 80
4.1.2 Predictive Value of a Positive or Negative 82
4.1.3 Sensitivity, Specificity, and Predictive Values with Clustered Binary-scale Data 84
4.1.4 Likelihood Ratio 86
4.1.5 Odds Ratio 88
4.2 Ordinal-scale Data 89
4.2.1 Empirical ROC Curve 90
4.2.2 Fitting a Smooth Curve 90
4.2.3 Estimation of Sensitivity at a Particular FPR 95
4.2.4 Area and Partial Area Under the ROC Curve (Parametric Methods) 97
4.2.5 ci Estimation 99
4.2.6 Area and Partial Area Under the ROC Curve (Nonparametric Methods) 102
4.2.7 Nonparametric Analysis of Clustered Data 105
4.2.8 Degenerate Data 106
4.2.9 Choosing Between Parametric, Semi-parametric, and Nonparametric Methods 108
4.3 Continuous-scale Data 108
4.3.1 Empirical ROC Curve 109
4.3.2 Fitting a Smooth ROC Curve - Parametric, Semi-parametric, and Nonparametric Methods 110
4.3.3 Confidence Bands Around the Estimated ROC Curve 115
4.3.4 Area and Partial Area Under the ROC Curve - Parametric, Nonparametric, and Semi-parametric Methods 116
4.3.5 CIs for the Area Under the ROC Curve 117
4.3.6 Fixed FPR - Sensitivity and the Decision Threshold 119
4.3.7 Choosing the Optimal Operating Point and Decision Threshold 122
4.3.8 Choosing Between Parametric, Semi-parametric, and Nonparametric Methods 125
4.4 Testing the Hypothesis that the ROC Curve Area or Partial Area Is a Specific Value 126
4.4.1 Testing Whether MRA Has Any Ability to Detect Significant Carotid Stenosis 127
5 Comparing the Accuracy of Two Diagnostic Tests 129
5.1 Binary-scale Data 130
5.1.1 Sensitivity and Specificity 130
5.1.2 Sensitivity and Specificity of Clustered Binary Data 132
5.1.3 Predictive Probability of a Positive or Negative 134
5.2 Ordinal- and Continuous-scale Data 136
5.2.1 Testing the Equality of Two ROC Curves 137
5.2.2 Comparing ROC Curves at a Particular Point 140
5.2.3 Determining the Range of FPRs for Which TPRs Differ 141
5.2.4 Comparison of the Area or Partial Area 143
5.3 Tests of Equivalence 148
5.3.1 Testing Whether ROC Curve Areas Are Equivalent: Case Study 3 150
6 Sample Size Calculations 153
6.1 Studies Estimating the Accuracy of a Single Test 153
6.1.1 Sample Size Calculations for Estimating Sensitivity and/or Specificity - Case Study 1 153
6.1.2 Sample Size for Estimating the Area Under the ROC Curve - Case Study 2 155
6.1.3 Studies with Clustered Data 157
6.1.4 Testing the Hypothesis That the ROC Area Is Equal to a Particular Value 158
6.1.5 Sample Size for Estimating Sensitivity at Fixed FPR - Case Study 2 158
6.1.6 Sample Size for Estimating the Partial Area Under the ROC Curve - Case Study 2 160
6.2 Sample Size for Detecting a Difference in Accuracies of Two Tests 161
6.2.1 Sample Size Software 161
6.2.2 Sample Size for Comparing Tests' Sensitivity and/or Specificity - Case Study 1 161
6.2.3 Sample Size for Comparing Tests' Positive and Negative Predictive Values - Case Study 1 163
6.2.4 Sample Size for Comparing Tests' Area Under the ROC Curve - Case Study 2 164
6.2.5 Sample Size for Comparing Tests with Clustered Data 165
6.2.6 Sample Size for Comparing Tests' Sensitivity at Fixed FPR - Case Study 2 166
6.2.7 Sample Size for Comparing Tests' Partial Area Under the ROC Curve - Case Study 2 167
6.3 Sample Size for Assessing Non-inferiority or Equivalency of Two Tests 169
6.4 Sample Size for Determining a Suitable Cutoff Value 172
6.5 Sample Size Determination for Multi-reader Studies 173
6.5.1 MRMC Sample Size Software 173
6.5.2 MRMC Sample Size Calculations with No Pilot Data 174
6.5.3 MRMC Sample Size Calculations with Pilot Data 179
6.6 Alternative to Sample Size Formulae 180
7 Introduction to Meta-analysis for Diagnostic Accuracy Studies 181
7.1 Objectives 182
7.2 Retrieval of the Literature 182
7.2.1 Literature Search: Meta-analysis of Ultrasound for PAD 186
7.3 Inclusion/Exclusion Criteria 186
7.3.1 Inclusion/Exclusion Criteria: Meta-analysis of Ultrasound for PAD 188
7.4 Extracting Information from the Literature 188
7.4.1 Data Abstraction: Meta-analysis of Ultrasound for PAD 190
7.5 Statistical Analysis 190
7.5.1 Binary-scale Data 190
7.5.2 Ordinal- or Continuous-scale Data 191
7.5.3 Area Under the ROC Curve 200
7.5.4 Other Methods 202
7.6 Public Presentation 202
7.6.1 Presentation of Results: Meta-analysis of Ultrasound for PAD 204
Part II Advanced Methods 205
8 Regression Analysis for Independent ROC Data 207
8.1 Four Clinical Studies 208
8.1.1 Surgical Lesion in a Carotid Vessel Example 208
8.1.2 Pancreatic Cancer Example 208
8.1.3 Hearing Test Example 208
8.1.4 Staging of Prostate Cancer Example 209
8.2 Regression Models for Continuous-scale Tests 210
8.2.1 Indirect Regression Models for ROC Curves 211
8.2.2 Direct Regression Models for ROC Curves 214
8.3 Regression Models for Ordinal-scale Tests 228
8.3.1 Indirect Regression Models for Latent Smooth ROC Curves 228
8.3.2 Direct Regression Model for Latent Smooth ROC Curves 230
8.3.3 Detection of Periprostatic Invasion with Ultrasound 232
8.4 Covariate AROC Curves of Continuous-scale Tests 233
9 Analysis of Multiple Reader and/or Multiple Test Studies 235
9.1 Studies Comparing Multiple Tests with Covariates 235
9.1.1 Two Clinical Studies 235
9.1.2 Indirect Regression Models for Ordinal-scale Tests 236
9.1.3 Direct Regression Models for Continuous-scale Tests 241
9.2 Studies with Multiple Readers and Multiple Tests 245
9.2.1 Three MRMC Studies 245
9.2.2 Statistical Methods for Analyzing MRMC Studies 246
9.2.3 Analysis of the Interstitial Disease Example 254
9.2.4 Comparisons Between MRMC Methods 254
10 Methods for Correcting Verification Bias 257
10.1 Examples 258
10.1.1 Hepatic Scintigraph 258
10.1.2 Screening Tests for Dementia Disorder Example 258
10.1.3 Fever of Uncertain Origin 259
10.1.4 CT and MRI for Staging Pancreatic Cancer Example 259
10.1.5 NACC MDS on AD 259
10.2 Impact of Verification Bias 260
10.3 A Single Binary-scale Test 261
10.3.1 Correction Methods Under the MAR Assumption 261
10.3.2 Correction Methods Without the MAR Assumption 263
10.3.3 Analysis of Hepatic Scintigraph Example, Continued 265
10.4 Correlated Binary-scale Tests 267
10.4.1 ml Approach Without Any Covariates 267
10.4.2 Analysis of Two Screening Tests for Dementia Disorder Example 272
10.4.3 ml Approach with Covariates 273
10.4.4 Analysis of Two Screening Tests for Dementia Disorder Example, Continued 275
10.5 A Single Ordinal-scale Test 276
10.5.1 ML Approach Without Covariates 276
10.5.2 Analysis of Fever of Uncertain Origin Example 280
10.5.3 ML Approach with Covariates 280
10.5.4 Analysis of New Screening Test for Dementia Disorder 284
10.6 Correlated Ordinal-scale Tests 286
10.6.1 Weighted Estimating Equation Approaches for Latent Smooth ROC Curves 287
10.6.2 Likelihood-based Approach for ROC Areas 291
10.6.3 Analysis of CT and MRI for Staging Pancreatic Cancer 295
10.7 Continuous-scale Tests 296
10.7.1 Estimation of ROC Curves and Their Areas Under the MAR Assumption 297
10.7.2 Estimation of ROC Curves and Areas Under a Non-MAR Process 303
11 Methods for Correcting Imperfect Gold Standard Bias 313
11.1 Examples 314
11.1.1 Binary Stool Test for Strongyloides Infection 314
11.1.2 Binary Tine Test for Tuberculosis 314
11.1.3 Binary-scale X-rays for Pleural Thickening 314
11.1.4 Bioassays for HIV 315
11.1.5 Ordinal-scale Evaluation by Pathologists for Detecting Carcinoma In Situ of the Uterine Cervix 315
11.1.6 Ordinal-scale and Continuous-scale MRA for Carotid Artery Stenosis 315
11.2 Impact of Imperfect Gold Standard Bias 315
11.3 One Single Binary Test in a Single Population 317
11.3.1 Conditions for Model Identifiability 318
11.3.2 The Frequentist-based ML Method Under an Identifiable Model 319
11.3.3 Bayesian Methods Under a Non-identifiable Model 320
11.3.4 Analysis of Strongyloides Infection Example 322
11.4 One Single Binary Test in G Populations 324
11.4.1 Estimation Methods 324
11.4.2 Tuberculosis Example 327
11.5 Multiple Binary Tests in One Single Population 329
11.5.1 Checking for Model Identifiability 329
11.5.2 ml Estimates Under the CIA 330
11.5.3 Assessment of Pleural Thickening Example 331
11.5.4 ml Approaches Under Identifiable Conditional Dependence Models 331
11.5.5 Bioassays for HIV Example 336
11.5.6 Bayesian Methods Under Conditional Dependence Models 339
11.5.7 Analysis of the MRA for Carotid Stenosis Example 340
11.6 Multiple Binary Tests in G Populations 341
11.6.1 ml Approaches Under the CIA 342
11.6.2 ml Approach Without the CIA Assumption 343
11.7 Multiple Ordinal-scale Tests in One Single Population 343
11.7.1 Nonparametric Estimation of ROC Curves Under the CIA 343
11.7.2 Estimation of ROC Curves Under Some Conditional Dependence Models 345
11.7.3 Analysis of Ordinal-scale Tests for Detecting Carcinoma In Situ of the Uterine Cervix 346
11.8 Multiple-scale Tests in One Single Population 347
11.8.1 Reanalysis of the Accuracy of Continuous-scale MRA for Detection of Significant Carotid Stenosis 351
12 Location-specific ROC Methods for Diagnostic Imaging 353
12.1 Examples 353
12.1.1 Example One: A Clinical Study of Computer-aided Detection of Mammographic Masses 353
12.1.2 Example Two: A Clinical Study of Pulmonary Computer-aided Diagnosis Medical Software 354
12.1.3 Example Three: A Clinical Study of 3D Magnetic Resonance Angiography 354
12.2 LROC Approach 355
12.2.1 Swensson's Parametric Model 356
12.2.2 Nonparametric LROC Approach 358
12.2.3 Analysis of Example One 359
12.3 FROC Approach 360
12.3.1 FROC-type Curves 361
12.3.2 Radiological Search Model 367
12.3.3 Resampling Methods 368
12.3.4 FROC Analysis Method in MRMC Study 369
12.3.5 Analysis of Example Two 373
12.4 ROI Approach 377
12.4.1 Nonparametric ROI Analysis Method 377
12.4.2 iROI Paradigm 379
12.4.3 Analysis of Example Two 382
12.4.4 Analysis of Example Three 382
12.5 Comparison Between Location-specific ROC Methods 383
13 Technical Performance ("Accuracy") of Quantitative Imaging Biomarkers 385
13.1 Quantitative Imaging Biomarkers 385
13.1.1 Definitions 385
13.1.2 Technical Performance Characteristics 385
13.1.3 Illustrative Example of QIBs for Nonalcoholic Fatty Liver Disease 387
13.2 Technical Performance Characteristics of a QIB 387
13.2.1 Basic Model 387
13.2.2 Limits of Detection and Quantitation 388
13.3 Precision 389
13.3.1 Study Design for Measuring Precision of QIBs 389
13.3.2 Precision Metrics and Their Estimation 392
13.3.3 Sample Size Considerations for Precision Studies 396
13.3.4 Precision Profile 398
13.4 Bias and Linearity 398
13.4.1 Study Design for Assessing Bias 398
13.4.2 Bias Metrics and Their Estimation 401
13.4.3 Sample Size Considerations for Bias Studies 404
13.4.4 Bias Profile 404
13.5 Other Metrics of QIB Performance 405
13.5.1 Limits of Agreement 405
13.5.2 Coverage Probability 406
13.5.3 Total Deviation Index 406
13.5.4 Mean Squared Deviation 407
13.6 Clinical Performance 407
13.6.1 Integrated Biomarkers 408
13.6.2 Integral Biomarkers 409
13.6.3 Multi-parametric Applications 411
14 Medical Test Efficacy and Effectiveness 413
14.1 General Notation 414
14.2 Prognostic Effects 416
14.3 Predictive Effects 416
14.4 Test Strategies for Assigning Treatments 417
14.5 Explanatory Versus Pragmatic Trials of Tests 417
14.6 Explanatory Trial Designs 418
14.6.1 Stratified Design 418
14.6.2 Enrichment (Targeted) Design 418
14.6.3 Discordant Pairs Design 419
14.7 Pragmatic Trial Designs 420
14.7.1 Test Strategy Design 420
14.7.2 Comparative Test Strategy Design 422
14.8 Adaptive Treatment Strategy Trial Designs 423
14.9 Treatment Selection Tests 424
14.10 Follow-on Treatment Selection Tests 425
14.10.1 Bridging Studies 425
14.10.2 Concordance Study and Indirect Estimation of Drug Efficacy 427
14.11 Bibliographic Notes 428
14.11.1 Biomarkers and Markers 428
14.11.2 Prognostic and Predictive Factors 428
14.11.3 Medical Test Trial Design Literature 429
14.11.4 Stratified Trial Design 429
14.11.5 Enrichment Design 430
14.11.6 Causal Effects of Testing 430
15 Statistical Analysis for Meta-analysis 433
15.1 Binary-scale Data 433
15.1.1 Random Effects Model: Meta-analysis of Ultrasound for PAD 434
15.2 Ordinal- or Continuous-scale Data 435
15.2.1 Random Effects Model 435
15.2.2 Bivariate Approach 436
15.2.3 Binary Regression Model 438
15.2.4 Hierarchical SROC Curve 439
15.2.5 Other Methods 441
15.3 ROC Curve Area 441
15.3.1 EB Method: Meta-analysis of DST 443
15.4 Publication Bias 443
16 Risk Prediction 449
16.1 Risk Calculators 451
16.1.1 Breast Cancer Calculators 451
16.1.2 The Gail Model 452
16.1.3 Framingham Heart Study Risk Calculator 453
16.2 Calibration 454
16.2.1 Definitions of Calibration 454
16.2.2 Log-logistic Regression 455
16.2.3 Cox Proportional Hazards 456
16.2.4 Predictiveness Curve 457
16.2.5 Calibration Plot 458
16.2.6 Goodness-of-fit Tests 459
16.2.7 Testing for Equivalence 461
16.2.8 Brier Score 463
16.2.9 Model-based ROC Curve 463
16.3 Discrimination 464
16.3.1 Risk Distribution 465
16.3.2 Time-dependent Predictive Values 466
16.3.3 Time-dependent ROC Curves 467
16.4 Appendix 16-A: Survival Analysis 469
16.4.1 Structure of Survival Data 470
16.4.2 Basic Concepts in Survival Analysis 471
16.4.3 Estimation 472
16.5 Appendix 16-B: Survival Analysis for Competing Risks 475
16.5.1 Competing Risk Survival Analysis 475
16.5.2 Survival Analysis for Competing Risks 476
16.5.3 Estimation 478
16.5.4 Kaplan-Meier Estimator of Pure Risk 478
16.5.5 Aalen-Johansen Estimator of Absolute Risk 479
16.6 Appendix 16-C: Stochastic Processes for Survival Analysis 480
16.7 Appendix 16-D: Bibliographic Notes 481
16.7.1 Bibliographic Notes for Section 16.3 481
16.7.2 Bibliographic Notes for Section 16.5 481
Appendix-A: Case Studies and Chapter 8 Data 485
Appendix-B: Jackknife and Bootstrap Methods of Estimating Variances and Confidence Intervals 513
Bibliography 517
Index 555
