Advertisement

Math Matters: Why Positive Screening Rates Cannot Substitute for Prevalence

Published:November 06, 2021DOI:https://doi.org/10.1016/j.acap.2021.10.013

      Keywords

      Imagine this: concerned about postnatal depression, a pediatric practice begins a screening program that includes use of the Edinburgh Postnatal Depression Scale (EPDS). Diligently tracking results, the pediatricians find that 22.4% of individuals screened positive over the first year—markedly higher than a recent meta-analysis that estimated 9.3% prevalence in high-income countries like the US.
      • Woody CA
      • Ferrari AJ
      • Siskind DJ
      • et al.
      A systematic review and meta-regression of the prevalence and incidence of perinatal depression.
      Should they be alarmed that the test positivity rate is more than twice the estimated prevalence?
      The answer depends on whether the number of individuals in a population who score positive on a screening test in a given time period is comparable to the number who have a specific disorder or characteristic. Investigators often equate the two, but there is a problem—positive screen rates and prevalence are not the same. Having a disorder may contribute to a positive screening result, but other critical factors play a role as well.
      Consider the 2x2 table depicted in Figure 1. The upper left-hand corner—comprised of individuals who both have a disorder and screen positive (ie, true positive proportion)—contributes to both prevalence and total positive proportion. Here, prevalence and total positive proportion intersect. But prevalence also includes the lower left-hand corner—comprised of individuals who have the disorder yet screen negative (false negative proportion), whereas total positive proportion also includes the upper right hand corner—comprised of individuals who do not have the disorder yet screen positive (false positive proportion). If false positives outnumber false negatives, then total positive proportion will exceed prevalence. Conversely, if false negatives outnumber false positives, then total positive proportion will fall short of prevalence. The proportion of a population who test positive on a screening test is not necessarily a good approximation of prevalence.
      Figure 1
      Figure 1The difference between prevalence and total positive proportion on a screening questionnaire.
      Unfortunately, it is often the case that the only available evidence applicable to prevalence derives from questionnaires or other tests that lack perfect reliability, such as those used for screening. For some problems—such as adverse childhood experiences (ACEs)—nearly all evidence of prevalence relies on questionnaires that have been developed through population surveys and are now widely used for screening.
      • Barnett ML
      • Sheldrick RC
      • Liu S
      • et al.
      Implications of ACEs screening on behavioral health services: a scoping review and systems modeling analysis.
      Likewise, pediatricians screening for social determinants of health like food insecurity or housing instability may look to screening results to understand the level of need in their populations. Even when credible prevalence estimates have been published, as is the case for autism,
      • Baio J
      • Wiggins L
      • Christensen DL
      • et al.
      Prevalence of autism spectrum disorder among children aged 8 years—autism and developmental disabilities monitoring network, 11 sites, United States, 2014.
      providers may reasonably expect local variation in prevalence
      • Sheldrick RC
      • Carter AS
      State-level trends in the prevalence of Autism Spectrum Disorder (ASD) from 2000 to 2012: a reanalysis of findings from the autism and developmental disabilities network.
      and wonder whether national or state-level estimates apply to their specific patient population. In other cases, meta-analyses of prevalence studies using structured interviews may supplement their analyses with additional studies that include only screening data (such as the meta-analysis on perinatal depression cited above).
      • Woody CA
      • Ferrari AJ
      • Siskind DJ
      • et al.
      A systematic review and meta-regression of the prevalence and incidence of perinatal depression.
      To appropriately estimate prevalence using screening data, we need to know more about the screening test's error rates—that is, the false positive and false negative proportions. As depicted in Figure 1, these error rates are functions of sensitivity, specificity, and prevalence. Summing the top two cells, we can see that total positive proportion is governed by the following equation:
      totalpositiveproportion=sensitivity·prevalence+(1specificity)·(1prevalence)


      That is, the total positive proportion equals the proportion of respondents with a disorder who screen positive (ie, sensitivity*prevalence) plus the proportion of respondents without a disorder (ie, 1-prevalence) who also screen positive (ie, 1-specificity).  Now reconsider the pediatricians in our example of postnatal depression. What they know is that 22.4% of individuals screened positive on the EPDS over the first year. How should they interpret this fact? On the left-hand side of Figure 2 (scenario A), the false positive and false negative proportions are calculated based on estimates of sensitivity and specificity drawn from a recent meta-analysis of the EPDS.
      • Levis B
      • Negeri Z
      • Sun Y
      • et al.
      Accuracy of the Edinburgh Postnatal Depression Scale (EPDS) for screening to detect major depression among pregnant and postpartum women: systematic review and meta-analysis of individual participant data.
      The result is a prevalence of 9.3%
      • Woody CA
      • Ferrari AJ
      • Siskind DJ
      • et al.
      A systematic review and meta-regression of the prevalence and incidence of perinatal depression.
      —exactly the same as the estimate reported in the literature!
      Figure 2
      Figure 2Two (of many) scenarios consistent with a total positive proportion of 22.4%.
      Should the pediatricians be reassured that prevalence in their patient population is similar to the general population? Only if they trust the external validity of published results and therefore believe that sensitivity and specificity estimates generalize to their clinic and patient population. Unfortunately, the meta-analysis noted significant heterogeneity of sensitivity and specificity among studies,
      • Levis B
      • Negeri Z
      • Sun Y
      • et al.
      Accuracy of the Edinburgh Postnatal Depression Scale (EPDS) for screening to detect major depression among pregnant and postpartum women: systematic review and meta-analysis of individual participant data.
      suggesting that the values reported do not apply equally to all cases. For example, what if the pediatricians suspect that their population worries about stigma more than most perinatal individuals and are therefore less likely to disclose symptoms whether or not they have depression? If so, they might hypothesize that sensitivity is lower and that specificity is higher than published estimates. If this were the case, what would it mean if 22.4% of individuals screened positive? The right-hand side of Figure 2 (scenario B) suggests one possibility, reflecting an actual prevalence of 20% rather than 9.3%.
      Bottom line: estimates of prevalence are influenced by assumptions regarding the accuracy of the assessment tools used. In the case of more rigorous diagnostic tools, it is common to assume that sensitivity and specificity are high enough that we can accept their results as accurate and true, as is the case in most prevalence studies. In contrast, we know that screeners are not perfectly accurate (why else would they ever lead to diagnostic evaluations?), so we should account for their sensitivity and specificity when estimating prevalence. Unfortunately, there are many reasons why screening accuracy in practice may diverge from published results. Some reasons have to do with the tests themselves, such as the quality and comparability of translated questionnaires used with non-English speaking patients
      • Ayalew B
      • Dawson-Hahn E
      • Cholera R
      • et al.
      The health of children in immigrant families: key drivers and research gaps through an equity lens.
      and choices regarding screening thresholds (ie, cut scores), which are set to optimize the balance between sensitivity and specificity in a clinical process.
      • Sheldrick RC
      • Garfinkel D
      Is a positive developmental-behavioral screening score sufficient to justify a referral? A review of evidence and theory.
      Other reasons have to do with families and their interactions with the medical system, such as their ability and willingness to disclose symptoms and their trust in medical providers. All of these factors may result in people responding differently to screening questionnaires in different contexts, thus resulting in differences in sensitivity and specificity (which are too often assumed to be stable test characteristics).
      Returning to ACEs as an example, recent articles in this journal advance discussion about the relationship between research, practice, and policy,
      • Conn AM
      • Szilagyi M
      • Forkey H
      Adverse childhood experience and social risk: pediatric practice and potential.
      for example how to effectively communicate about evidence to influence policy.
      • Purtle J
      • Nelson KL
      • Srivastav A
      • et al.
      Perceived persuasiveness of evidence about adverse childhood experiences: results from a national survey.
      How can such discussions make best use of available screening data? For example, how should we interpret data from state-wide implementation of ACEs screening in California which reveals notable regional differences in positive screening results?

      Fact Sheet: MediCal Claims for ACEs screening. Available at: https://www.acesaware.org/wp-content/uploads/2021/03/ACEs-Claims-Data-Fact-Sheet-FINAL-2-25-21_a11y.pdf. 2021. Accessed March 30, 2021.

      Our answer: do not equate total positive proportion with prevalence; instead, understand the math that governs screening results and carefully consider the assumptions regarding sensitivity and specificity that underlie it.

      References

        • Woody CA
        • Ferrari AJ
        • Siskind DJ
        • et al.
        A systematic review and meta-regression of the prevalence and incidence of perinatal depression.
        J. Affect. Disord. 2017; 219: 86-92
        • Barnett ML
        • Sheldrick RC
        • Liu S
        • et al.
        Implications of ACEs screening on behavioral health services: a scoping review and systems modeling analysis.
        Am Psychol. 2021; 76: 364-378
        • Baio J
        • Wiggins L
        • Christensen DL
        • et al.
        Prevalence of autism spectrum disorder among children aged 8 years—autism and developmental disabilities monitoring network, 11 sites, United States, 2014.
        MMWR CDC Surveill Summ. 2018; 67: 1
        • Sheldrick RC
        • Carter AS
        State-level trends in the prevalence of Autism Spectrum Disorder (ASD) from 2000 to 2012: a reanalysis of findings from the autism and developmental disabilities network.
        J Autism Dev Disord. 2018; 48: 3086-3092
        • Levis B
        • Negeri Z
        • Sun Y
        • et al.
        Accuracy of the Edinburgh Postnatal Depression Scale (EPDS) for screening to detect major depression among pregnant and postpartum women: systematic review and meta-analysis of individual participant data.
        BMJ. 2020; 371: m4022
        • Ayalew B
        • Dawson-Hahn E
        • Cholera R
        • et al.
        The health of children in immigrant families: key drivers and research gaps through an equity lens.
        Acad Pediatr. 2021; 21: 777-792
        • Sheldrick RC
        • Garfinkel D
        Is a positive developmental-behavioral screening score sufficient to justify a referral? A review of evidence and theory.
        Acad Pediatr. 2017; 17: 464-470
        • Conn AM
        • Szilagyi M
        • Forkey H
        Adverse childhood experience and social risk: pediatric practice and potential.
        Acad Pediatr. 2020; 20: 573-574
        • Purtle J
        • Nelson KL
        • Srivastav A
        • et al.
        Perceived persuasiveness of evidence about adverse childhood experiences: results from a national survey.
        Acad Pediatr. 2020; 21: 529-533
      1. Fact Sheet: MediCal Claims for ACEs screening. Available at: https://www.acesaware.org/wp-content/uploads/2021/03/ACEs-Claims-Data-Fact-Sheet-FINAL-2-25-21_a11y.pdf. 2021. Accessed March 30, 2021.