Advertisement

The Contribution of Reminder-Recall to Vaccine Delivery Efforts: A Narrative Review

  • Allison Kempe
    Correspondence
    Address correspondence to Allison Kempe, MD MPH, Adult and Child Consortium for Health Outcomes Research and Delivery Science (ACCORDS), University of Colorado School of Medicine and Children's Hospital Colorado, Mail Stop F443 | 13199 E. Montview Blvd, Suite 300, Aurora, CO 80045
    Affiliations
    Adult and Child Consortium for Health Outcomes Research and Delivery Science (ACCORDS), University of Colorado School of Medicine and Children's Hospital Colorado (A Kempe), Aurora, Colo

    Department of Pediatrics, University of Colorado School of Medicine (A Kempe), Aurora, Colo
    Search for articles by this author
  • Melissa S. Stockwell
    Affiliations
    Division of Child and Adolescent Health, Department of Pediatrics, Columbia University Irving Medical Center (MS Stockwell), New York, NY

    Department of Population and Family Health, Columbia University Irving Medical Center (MS Stockwell), New York, NY
    Search for articles by this author
  • Peter Szilagyi
    Affiliations
    Department of Pediatrics, UCLA Mattel Children's Hospital, University of California at Los Angeles (P Szilagyi)
    Search for articles by this author

      Abstract

      Reminders, alerting patients to the need for vaccines that will be due in the future, and recall messages, informing patients about vaccines that are overdue, have been shown to improve immunization rates for children and adolescents in numerous systematic reviews. Therefore, reminder and recall interventions (R/R) are recommended by the Task Force on Community Preventive Services for increasing immunization rates on the basis of strong evidence. R/R messages can be delivered by mail (letter or postcard), via personal or auto-dialer phone calls, by text or e-mail or via patient-portals and can simply be alerts to action or can include educational material with the aim of motivating patients to seek vaccination. R/R has also been shown to be a relatively low-cost intervention with high cost-effectiveness compared with other recommended strategies. However, although R/R as a strategy is consistently effective and cost-effective overall, there is wide variation in the impact of R/R by 1) modality of how it is delivered, 2) the targeted vaccine, 3) the age group, and 4) whether the R/R is conducted centrally by a health system or Immunization Information System or by individual practices. This narrative review summarizes the literature about effectiveness of R/R within each of these categories. We also discuss limitations of R/R, with a focus on the potential impact of parental vaccine hesitancy in blunting its effectiveness and problems with data integrity, on which R/R relies. We also discuss challenges to sustaining R/R efforts, including potential methods of funding for R/R efforts.

      Keywords

      Overview of Reminder/Recall

      Reminder/recall (R/R) of patients is an evidence-based strategy that has been shown to improve immunization rates for children and adolescents.
      • Jacobson Vann JC
      • Jacobson RM
      • Coyne-Beasley T
      • et al.
      Patient reminder and recall interventions to improve immunization rates.
      Reminder messages alert patients to the need for vaccines that will be due in the future and recall messages inform patients about vaccines that are overdue; however, the terms are generally combined because efforts to remind and recall usually occur together. Important domains characterizing R/R efforts are summarized in Table 1. R/R can be delivered by mail (letter or postcard), via personal or auto-dialer phone calls, by text or e-mail or via patient-portals. Such messages can simply be alerts to action or can include educational material that aims to increase vaccine health literacy with the aim of motivating patients to seek vaccination. The number of R/R messages sent varies, but is often between one and 3 messages, optimally with removal of patients who have received vaccines within each R/R cycle. One of the most important variations in implementation of R/R efforts is the unit of delivery, most commonly either at the clinical practice level or via an entity using a centralized data system, such as the public health department using an immunization information system (IIS) or a health system using a centralized patient database.
      Table 1Key Domains Characterizing Reminder/Recall (R/R) Efforts
      R/R CharacteristicOptions
      Delivery modalityMail (letter or postcard), Phone (auto-dialer or personal call), text, e-mail, patient portals
      FrequencyUsually 1, 2 or 3 reminders with responders removed between efforts
      Target vaccine(s)Often routine childhood, adolescent, or adult vaccines but may target an individual vaccine such as HPV or Influenza vaccines
      Age groupOften 20-35 months, school-age, adolescent, all adults, >65 year old adults
      ContentSimple cue to action or can include educational or motivational content
      Unit of deliveryOften individual practices (practice-based R/R) or health care systems or public health departments using an Immunization Information System (centralized R/R)
      Table 2Effectiveness of Patient Reminder/Recall (R/R) Using Different Modalities Compared With No R/R for Receipt of Immunizations
      Intervention typeRelative Risk (95% Confidence Interval)Certainty of Evidence (GRADE)
      Patient postcard reminder or recallRR 1.18 (1.08–1.30)High
      Patient text message reminder or recallRR 1.29 (1.15–1.44)High
      Patient auto-dialer message reminder or recallRR 1.17 (1.03–1.32)High
      Patient reminder or recall summary measureRR 1.28 (1.23–1.35)Moderate
      Patent telephone reminder or recallRR 1.75 (1.20–2.54)Moderate
      Patient letter reminder or recallRR 1.29 (1.21–1.38)Moderate
      Combination of patient reminder or recall with outreach interventionRR 1.22 (1.10–1.35)High
      Combination of patient mail and telephone reminder or recallRR 1.28 (1.14–1.45)Moderate
      Combination of patient reminder or recall with provider reminder interventionRR 2.91 (2.67–3.19)Moderate
      Adapted from: Jacobson Vann et al.
      • Jacobson Vann JC
      • Jacobson RM
      • Coyne-Beasley T
      • et al.
      Patient reminder and recall interventions to improve immunization rates.
      Table 3Effectiveness of Patient Reminder/Recall (R/R) for Different Age Groups and Vaccines Compared With No R/R for Receipt of Immunizations
      Type of ImmunizationRelative Risk (95% Confidence Interval)Certainty of Evidence (GRADE)
      Childhood immunizationsRR 1.22 (1.15–1.29)High
      Adolescent immunizationsRR 1.29 (1.17–1.42)High
      Childhood influenza immunizationsRR 1.51 (1.14–1.99)Moderate
      Adult immunizations—Other than influenza or travel (‘Other adult’)RR 2.08 (0.91–4.78)Low
      Adult influenza immunizationsRR 1.29 (1.17–1.43)Moderate
      Adapted from: Jacobson Vann.
      • Jacobson Vann JC
      • Jacobson RM
      • Coyne-Beasley T
      • et al.
      Patient reminder and recall interventions to improve immunization rates.

      Effectiveness of R/R

      The effectiveness of R/R as a general strategy has been well-documented in numerous studies summarized in multiple Cochrane reviews.
      • Jacobson Vann JC
      • Jacobson RM
      • Coyne-Beasley T
      • et al.
      Patient reminder and recall interventions to improve immunization rates.
      • Jacobson Vann JC
      • Szilagyi P
      Patient reminder and patient recall systems to improve immunization rates.
      • Szilagyi PG
      • Bordley C
      • Vann JC
      • et al.
      Effect of patient reminder/recall interventions on immunization rates: a review.
      • Szilagyi P
      • Vann J
      • Bordley C
      • et al.
      Interventions aimed at improving immunization rates.
      It is recommended by the Task Force on Community Preventive Services as a method for increasing immunization rates

      Community Preventive Services Task Force. The Community Guide: Vaccination Programs: Client Reminder and Recall Systems. Centers for Disease Control and Prevention. Available at:https://www.thecommunityguide.org/findings/vaccination-programs-client-reminder-and-recall-systems. Accessed October 21, 2019.

      on the basis of strong evidence. The 2018 Cochrane review,
      • Jacobson Vann JC
      • Jacobson RM
      • Coyne-Beasley T
      • et al.
      Patient reminder and recall interventions to improve immunization rates.
      included 75 studies conducted in 10 countries, with the majority of studies being in the US. On average, R/R improved the proportion of patients who received immunizations by roughly 8 percentage points (ranging from 5-20 percentage points) with a risk ratio (RR) of 1.28 (95% confidence intervals [CI] 1.23–1.35). All the Cochrane reviews have been fairly consistent in their overall findings.
      • Jacobson Vann JC
      • Jacobson RM
      • Coyne-Beasley T
      • et al.
      Patient reminder and recall interventions to improve immunization rates.
      • Jacobson Vann JC
      • Szilagyi P
      Patient reminder and patient recall systems to improve immunization rates.
      • Szilagyi PG
      • Bordley C
      • Vann JC
      • et al.
      Effect of patient reminder/recall interventions on immunization rates: a review.
      • Szilagyi P
      • Vann J
      • Bordley C
      • et al.
      Interventions aimed at improving immunization rates.
      However, although R/R as a strategy is consistently effective overall, there is wide variation in the impact of R/R by 1) modality of how it is delivered, 2) focus of R/R (by age or specific vaccine) and 3) whether the R/R is conducted centrally by a system or IIS or by individual practices. Therefore, we will discuss the impact of R/R within each of these categories.

       Effectiveness of R/R by Delivery Modality

      All modes of delivery were found by the Cochrane review to be effective, although there was variability in the quality of the studies supporting the evidence. Text messaging showed the highest level of effectiveness for a single reminder method based on a high certainty of evidence, while patient telephone R/R showed the largest effect based on moderate certainty of evidence. R/R combining different modalities generally showed similar levels of effectiveness as single methods. However, there was substantial variation in effectiveness of different modalities by study, reflecting variation in the number of messages used in different protocols, how different modalities were combined, and the populations being targeted. As an example, although the literature shows substantial effectiveness of text messaging among many patient populations,
      • Stockwell MS
      • Kharbanda EO
      • Martinez RA
      • et al.
      Text4Health: impact of text message reminder–recalls for pediatric and adolescent immunizations.
      • Hofstetter AM
      • DuRivage N
      • Vargas CY
      • et al.
      Text message reminders for timely routine MMR vaccination: a randomized controlled trial.
      • O'Leary ST
      • Lee M
      • Lockhart S
      • et al.
      Effectiveness and cost of bidirectional text messaging for adolescent vaccines and well care.
      • Mekonnen ZA
      • Gelaye KA
      • Were MC
      • et al.
      Effect of mobile text message reminders on routine childhood vaccination: a systematic review and meta-analysis.
      it has been less effective in other studies involving populations where texting from a health provider is novel, where mobile phones are not ubiquitous, or among young adult populations.
      • Richman AR
      • Maddy L
      • Torres E
      • et al.
      A randomized intervention study to evaluate whether electronic messaging can increase human papillomavirus vaccine completion and knowledge among college students.
      • Patel A
      • Stern L
      • Unger Z
      • et al.
      Staying on track: a cluster randomized controlled trial of automated reminders aimed at increasing human papillomavirus vaccine completion.
      • Berenson AB
      • Rahman M
      • Hirth JM
      • et al.
      A human papillomavirus vaccination program for low-income postpartum women.
      • LeBaron CW
      • Starnes DM
      • Rask KJ.
      The impact of reminder-recall interventions on low vaccination coverage in an inner-city population.
      • Hambidge SJ
      • Davidson AJ
      • Phibbs SL
      • et al.
      Strategies to improve immunization rates and well-child care in a disadvantaged population: a cluster randomized controlled trial.
      • Kempe A
      • Lowery NE
      • Pearson KA
      • et al.
      Immunization recall: effectiveness and barriers to success in an urban teaching clinic.
      • Irigoyen MM
      • Findley S
      • Wang D
      • et al.
      Challenges and successes of immunization registry reminders at inner-city practices.
      Studies that combined patient R/R with other interventions (such as prompts to healthcare providers at the time of a visit, or outreach to patients using outreach workers) appear to have a greater impact on vaccination rates.
      • Rodewald LE
      • Szilagyi PG
      • Humiston SG
      • et al.
      A randomized study of tracking with outreach and provider prompting to improve immunization coverage and primary care.
      • Szilagyi PG
      • Humiston SG
      • Gallivan S
      • et al.
      Effectiveness of a citywide patient immunization navigator program on improving adolescent immunizations and preventive care visit rates.
      • Hambidge SJ
      • Phibbs SL
      • Chandramouli V
      • et al.
      A stepped intervention increases well-child care and immunization rates in a disadvantaged population.
      However, the systematic review could not assess many of these multi-component studies quantitatively because of their heterogeneity in modalities and their inability to distinguish the impact of patient R/R from the impact of other interventions.
      • Jacobson Vann JC
      • Jacobson RM
      • Coyne-Beasley T
      • et al.
      Patient reminder and recall interventions to improve immunization rates.
      The review did evaluate the combination of R/R with outreach (generally targeting low-income populations), and noted improved vaccination rates with high certainty of evidence.
      Variation in the modalities of R/R that patients or parents prefer to receive may play a role in their responsiveness to the R/R intervention. Previous survey data demonstrated different preferences for R/R modalities between urban versus rural populations, within different socioeconomic strata and by type of health care delivery facility, especially regarding text messaging.
      • Saville AW
      • Beaty B
      • Dickinson LM
      • et al.
      Novel immunization reminder/recall approaches: rural and urban differences in parent perceptions.
      • Albright K
      • Hurley LP
      • Lockhart S
      • et al.
      Attitudes about adult vaccines and reminder/recall in a safety net population.
      • Hofstetter AM
      • Vargas CY
      • Kennedy A
      • et al.
      Parental and provider preferences and concerns regarding text message reminder/recall for early childhood vaccinations.
      • Clark SJ
      • Butchart A
      • Kennedy A
      • et al.
      Parents' experiences with and preferences for immunization reminder/recall technologies.
      However, given the changing landscape of digital technology and health care delivery in general, it is likely that patient preferences will be a moving target. There is also some evidence that allowing patients to choose the method of R/R, “preference-based R/R” can be more effective than strategies in which the method is dictated. For example, one study using preference-based R/R showed substantially higher effect sizes than seen in most previous R/R trials for any vaccine, with an absolute increase of 25 percentage points for HPV completion (63% vs 38%) compared with no R/R.
      • Kempe A
      • O'Leary ST
      • Shoup JA
      • et al.
      Parental choice of recall method for HPV vaccination: a pragmatic trial.

       Effectiveness of R/R Within Different Age Groups or for Different Vaccines

      R/R has been shown to be effective for increasing vaccination rates across the age spectrum, although there is variability in effectiveness by type of vaccine and age group. In the 2018 Cochrane review, risk differences averaged 8 absolute percentage points between R/R and control (23 studies) for routine childhood vaccinations and 7 percentage points (10 studies) for adolescent vaccinations. The data regarding effectiveness for childhood influenza suggested relatively large effect sizes, 17-26 absolute percentage points higher than controls (5 studies); however 4 of the 5 studies were among children with high-risk medical conditions. The one study that focused on healthy children
      • Kempe A
      • Daley MF
      • Barrow J
      • et al.
      Implementation of universal influenza immunization recommendations for healthy young children: results of a randomized, controlled trial with registry-based recall.
      found an increase of 4.4 absolute percentage points (62.4% vs 58.0%) among children 6-23 months of age. Similarly, another more recent trial not included in the Cochrane review focusing on R/R for healthy children of all age groups and using text reminders, found an absolute difference of 3.7% (43.6% vs 39.9%).
      • Stockwell MS
      • Kharbanda EO
      • Martinez RA
      • et al.
      Effect of a text messaging intervention on influenza vaccination in an urban, low-income pediatric and adolescent population: a randomized controlled trial.
      Therefore, the effect size for R/R for influenza vaccine for healthy children may be lower than the effect for children with chronic conditions and also lower compared with other childhood vaccines.

       Effectiveness of Centralized vs Practice-Based R/R

      There are 2 important limitations to the literature on patient R/R for vaccinations. First, many studies have been done in single practices or small numbers of practices. Second, the majority of studies included in Cochrane reviews of the effectiveness of R/R were based on research studies done with the assistance of an outside research study team and few measured the sustainability of R/R after such a trial. Previous national data suggested that only 16% of providers were doing any type of R/R
      • Tierney CD
      • Yusuf H
      • McMahon SR
      • et al.
      Adoption of reminder and recall messages for immunizations by pediatricians and public health clinics.
      on a routine basis and it is unknown how many are using an IIS to do so. Identified barriers to practice-based R/R cited by providers have included insufficient staff time and competing demands of primary care, staff turnover, costs related to mailings or telephone call reminders and the lack of computerized systems to readily identify patients who need immunizations.
      • Tierney CD
      • Yusuf H
      • McMahon SR
      • et al.
      Adoption of reminder and recall messages for immunizations by pediatricians and public health clinics.
      ,
      • Saville AW
      • Albright K
      • Nowels C
      • et al.
      Getting under the hood: exploring issues that affect provider-based recall using an immunization information system.
      However, a number of studies have shown that practice-based R/R is rarely implemented or sustained even when practices are offered training, have access to an IIS, and are provided with technical assistance and funding to cover their costs.
      • Saville AW
      • Albright K
      • Nowels C
      • et al.
      Getting under the hood: exploring issues that affect provider-based recall using an immunization information system.
      • Kempe A
      • Saville A
      • Dickinson LM
      • et al.
      Population-based versus practice-based recall for childhood immunizations: a randomized controlled comparative effectiveness trial.
      • Kempe A
      • Saville AW
      • Dickinson LM
      • et al.
      Collaborative centralized reminder/recall notification to increase immunization rates among young children: a comparative effectiveness trial.
      To address the challenges of cost, sustainability, and scalability, centralized R/R has been studied as an alternative to practice-based R/R and as a mechanism to scale-up patient R/R. Most often, this has taken the form of state or county health departments conducting R/R on a population level using a state or regional IIS. Centralized R/R not only has the advantage of unburdening practices, but also has the potential to reach children who are in the IIS but not currently affiliated with a practice site. A recent survey of 57 of the 62 existing IIS programs across the country demonstrated that 61% had conducted centralized R/R for counties or geographic areas, 84% focused on the childhood series, 70% on HPV, 61% on other adolescent vaccines, and 35% for childhood influenza.
      • Saville AW
      • Gurfinkel D
      • Beaty BL
      • et al.
      The potential for centralized reminder/recall to increase immunization rates: a national survey of immunization information systems (IIS) managers.
      There have been several large trials comparing centralized IIS-based R/R by public health departments to practice-based approaches for increasing routine childhood vaccines among young children demonstrating significantly higher rates at lower cost achieved with the centralized approach. Relative risks estimates for the centralized compared with the practice-based R/R approach among 19-35 month children in one study were 1.23 (95% CI 1.10, 1.37) for becoming up-to-date (UTD) and 1.26 (95% CI 1.115–1.38) for receipt of any needed vaccine.
      • Kempe A
      • Saville A
      • Dickinson LM
      • et al.
      Population-based versus practice-based recall for childhood immunizations: a randomized controlled comparative effectiveness trial.
      In a different study adjusted odd ratios were 1.34 (95% CI 1.09–1.64) for becoming UTD and 1.31 (95% CI 1.16–1.48) for receipt of a needed vaccine.
      • Kempe A
      • Saville AW
      • Dickinson LM
      • et al.
      Collaborative centralized reminder/recall notification to increase immunization rates among young children: a comparative effectiveness trial.
      Particularly notable were the substantially lower cost of centralized versus practice-based approaches, $17 versus $62 per child brought UTD in the first study
      • Kempe A
      • Saville A
      • Dickinson LM
      • et al.
      Population-based versus practice-based recall for childhood immunizations: a randomized controlled comparative effectiveness trial.
      and $24.72 versus $124.45 per child brought UTD in the second.
      • Kempe A
      • Saville AW
      • Dickinson LM
      • et al.
      Collaborative centralized reminder/recall notification to increase immunization rates among young children: a comparative effectiveness trial.
      Other studies of centralized R/R conducted within systems of care that have centralized databases, including Medicaid HMOs,
      • Szilagyi PG
      • Albertin C
      • Humiston SG
      • et al.
      A randomized trial of the effect of centralized reminder/recall on immunizations and preventive care visits for adolescents.
      ,
      • Rand CM
      • Brill H
      • Albertin C
      • et al.
      Effectiveness of centralized text message reminders on human papillomavirus immunization coverage for publicly insured adolescents.
      county Medicaid programs,
      • Dombkowski KJ
      • Harrington LB
      • Dong S
      • et al.
      Seasonal influenza vaccination reminders for children with high-risk conditions: a registry-based randomized trial.
      networks of community health centers,
      • Stockwell MS
      • Kharbanda EO
      • Martinez RA
      • et al.
      Text4Health: impact of text message reminder–recalls for pediatric and adolescent immunizations.
      or Kaiser HMOs
      • Chao C
      • Preciado M
      • Slezak J
      • et al.
      A randomized intervention of reminder letter for human papillomavirus vaccine series completion.
      have shown similar success with centralized R/R for a variety of vaccines.
      Surprisingly, the effects of centralized R/R approaches for increasing rates of influenza for children and HPV vaccines have been more modest. Two recent large trials examined the effectiveness of IIS-based R/R by State Public Health Departments in both New York State and Colorado compared with usual care.
      • Kempe A
      • Saville AW
      • Albertin C
      • et al.
      Centralized reminder/recall to increase influenza vaccination rates: a two-state pragmatic randomized trial.
      • Szilagyi PG
      • Albertin CS
      • Saville AW
      • et al.
      Effect of state immunization information system based reminder/recall for influenza vaccinations: a randomized trial of autodialer, text, and mailed messages.
      • Szilagyi P
      • Albertin C
      • Gurfinkel D
      • et al.
      Effect of state immunization information system centralized reminder and recall on HPV vaccination rates.

      Gurfinkel D, Kempe A, Albertin C, et al. Centralized reminder/recall of HPV vaccination: findings from two states: a randomized clinical trial. J Adolesc Health. In press.

      Effects of IIS-based centralized R/R for increasing influenza rates among children in 2 studies were modest, with maximum differences in receipt of influenza vaccine of 1.7 percentage points (25.3% vs 23.6%) between R/R and control and adjusted risk ratios varying from 1.05 to 1.06 across the 2 states. Similarly, IIS-based centralized R/R for increasing HPV vaccine initiation showed a maximum differences of 1.3 absolute percentages points (33.5% vs 31.2%) in intervention versus controls in Colorado (maximum adjusted risk ratio between intervention and control arms of 1.07) and no significant effect in New York. The methods used in these IIS-based centralized R/R trials for influenza and HPV vaccines were very similar to the methods used by IIS-based centralized R/R trials focused on childhood vaccines; all trials tested multiple reminders and included some educational information within the R/R messages based upon components of the Health Belief Model. However, the trials focusing on childhood vaccines
      • Kempe A
      • Saville A
      • Dickinson LM
      • et al.
      Population-based versus practice-based recall for childhood immunizations: a randomized controlled comparative effectiveness trial.
      ,
      • Kempe A
      • Saville AW
      • Dickinson LM
      • et al.
      Collaborative centralized reminder/recall notification to increase immunization rates among young children: a comparative effectiveness trial.
      noted far larger impact than did the trials focused on influenza or HPV vaccine.
      • Kempe A
      • Saville AW
      • Albertin C
      • et al.
      Centralized reminder/recall to increase influenza vaccination rates: a two-state pragmatic randomized trial.
      • Szilagyi PG
      • Albertin CS
      • Saville AW
      • et al.
      Effect of state immunization information system based reminder/recall for influenza vaccinations: a randomized trial of autodialer, text, and mailed messages.
      • Szilagyi P
      • Albertin C
      • Gurfinkel D
      • et al.
      Effect of state immunization information system centralized reminder and recall on HPV vaccination rates.

      Gurfinkel D, Kempe A, Albertin C, et al. Centralized reminder/recall of HPV vaccination: findings from two states: a randomized clinical trial. J Adolesc Health. In press.

      These findings suggest that other factors that may not be as responsive to R/R efforts are playing a dominant role.

      Limitations of R/R

       R/R in the Face of Vaccine Hesitancy

      An important factor that may be responsible for the findings of the 2 IIS-based centralized R/R trials discussed above is whether vaccine hesitancy regarding influenza and HPV vaccines is blunting the effectiveness of R/R. Two recent national surveys of parents,
      • Kempe A
      • Saville AW
      • Albertin C
      • et al.
      Parental hesitancy about routine childhood and influenza vaccinations: a national survey.
      ,
      • Szilagyi PG
      • Albertin CS
      • Gurfinkel D
      • et al.
      Prevalence and characteristics of HPV vaccine hesitancy among parents of adolescents across the US.
      one focused on childhood vaccines and childhood influenza and the other on HPV examined the level of hesitancy toward these 3 types of vaccines using modified versions of the Vaccine Hesitancy Scale, developed by the World Health Organization. The prevalence of hesitancy as defined by this scale was 6% for routine childhood vaccination, 26% for childhood influenza vaccines and 23% for HPV vaccines. In the face of much higher levels of hesitancy, simply reminding or recalling parents for a vaccination may have minimal impact when parents have concerns about that vaccine. Therefore, hesitancy may explain the higher level of effectiveness of centralized R/R for routine childhood vaccines, in contrast to both HPV and influenza vaccines, due to much higher levels of hesitancy for HPV and influenza vaccines than for childhood vaccines.
      In the face of high vaccine hesitancy, R/R may need to do more than simply cue caregivers or patients about the need for a vaccine. As one example in a group of parents whose children were not yet vaccinated against influenza by November, interactive text messaging in which a parent could request further information via the text message was effective versus usual care while conventional text message reminders were not.
      • Hofstetter AM
      • Vargas CY
      • Camargo S
      • et al.
      Impacting delayed pediatric influenza vaccination: a randomized controlled trial of text message reminders.
      Message effectiveness may be further enhanced by including principles of behavioral economics such as message framing or utilizing social norms.
      • Jenssen BP
      • Buttenheim AM
      • Fiks AG.
      Using behavioral economics to encourage parent behavior change: opportunities to improve clinical effectiveness.
      However a better understanding of what principles work best for which vaccines may be needed since not all studies suggest these methods are effective for increasing vaccination.

      Lerner C, Albertin C, Casillas A, et al. Patient portal reminders for pediatric influenza vaccinations: a randomized clinical trial. Pediatrics. In press.

      The role of hesitancy is suggested by comparing the impact of reminders for initial versus second influenza vaccination among children eligible for 2 doses. Two studies noted relatively large impact for the second dose

      Lerner C, Albertin C, Casillas A, et al. Patient portal reminders for pediatric influenza vaccinations: a randomized clinical trial. Pediatrics. In press.

      ,
      • Stockwell MS
      • Hofstetter AM
      • DuRivage N
      • et al.
      Text message reminders for second dose of influenza vaccine: a randomized controlled trial.
      while several contemporary studies (and one using the same population) did not note impact for the first dose, presumably because of high parental hesitancy. These findings suggest parental/patient hesitancy plays a major role in modulating the impact of R/R.

       Potential Impact of R/R If Scaled Up

      It is important to note, however, that even small increases in vaccination rates such as those seen for centralized R/R for HPV or influenza may have substantial benefit at the population level. For example, a recent modeling study demonstrated that a 5% increase in influenza vaccine coverage among children could result in 282,000 fewer influenza infections and 1,440 fewer hospitalizations during a severe influenza season.
      • Hughes MM
      • Reed C
      • Flannery B
      • et al.
      Projected population benefit of increased effectiveness and coverage of influenza vaccination on influenza burden in the United States.
      Modeling for smaller effect sizes has not been done, but across populations even differences of 1-2% may be very important. Nonetheless, interventions to increase rates for vaccines associated with higher rates of hesitancy will, ideally require a multi-dimensional intervention rather than R/R alone.

       R/R and Data Quality

      There are other limitations of R/R that may limit its impact. Principal among these are the fact that R/R is only as good as the data driving it. Immunization records may be fragmented at any one site of care, especially in low-income populations or when immunizations are given in retail pharmacies, potentially leading to erroneous R/R based on incomplete records.
      • Yusuf H
      • Adams M
      • Rodewald L
      • et al.
      Fragmentation of immunization history among providers and parents of children in selected underserved areas.
      • Kolasa MS
      • Chilkatowsky AP
      • Clarke KR
      • et al.
      How complete are immunization registries? The Philadelphia story.
      • Bardenheier BH
      • Yusuf HR
      • Rosenthal J
      • et al.
      Factors associated with underimmunization at 3 months of age in four medically underserved areas.
      If an IIS is used for R/R, immunization data may be limited by underreporting of vaccines, as IIS reporting is not mandatory in many states. IISs often suffer from incorrect contact information, because such data are not necessarily updated on a regular basis. Despite the fact that real-time electronic interfaces between practice data and IISs (HL7) have recently increased,
      • Saville AW
      • Gurfinkel D
      • Beaty BL
      • et al.
      The potential for centralized reminder/recall to increase immunization rates: a national survey of immunization information systems (IIS) managers.
      contact information is frequently not uploaded automatically and may require administration of a vaccine to trigger updating. Data also suggest that parents may be more responsive to R/R that appears to come from their child's provider
      • Kempe A
      • Saville AW
      • Dickinson LM
      • et al.
      Collaborative centralized reminder/recall notification to increase immunization rates among young children: a comparative effectiveness trial.
      ,
      • Kempe A
      • Saville AW
      • Beaty B
      • et al.
      Centralized reminder/recall to increase immunization rates in young children: how much bang for the buck?.
      and they usually prefer personalized messages,
      • Hofstetter AM
      • Vargas CY
      • Kennedy A
      • et al.
      Parental and provider preferences and concerns regarding text message reminder/recall for early childhood vaccinations.
      therefore R/R messages should be personalized if at all possible.

      Cost and Cost-Effectiveness of R/R

      One of the most compelling aspects of R/R as an approach for increasing vaccination rates is its relative low cost and high-cost-effectiveness compared with other recommended strategies. A recent economic systematic review of the twelve interventions recommended by the Community Preventive Services Task Force to increase vaccination coverage
      • Jacob V
      • Chattopadhyay SK
      • Hopkins DP
      • et al.
      Increasing coverage of appropriate vaccinations: a community guide systematic economic review.
      compared the reach, cost and cost effectiveness of each strategy. Patient reminder systems were the lowest cost interventions to implement and demonstrated the lowest cost per additional person vaccinated of any intervention. The number of patients reached was highest for interventions that were conducted within systems rather than those conducted by individual practices. These findings additionally support the use of a centralized R/R approach using data from a larger system to increase sustainment and efficiency.

      Sustainability of R/R Efforts

      Despite all the data demonstrating effectiveness, cost-effectiveness and efficiency of R/R, a central dilemma remains. Whose responsibility is it to fund R/R efforts to assure they are sustainable? Even when the most efficient and least costly methods are used, such as IIS-based centralized R/R, there are some costs involved, primarily personnel costs. The current pandemic has highlighted the need for and critical importance of a strong public health infrastructure and since immunizations are one of the ten most important public health advances,
      Centers for Disease Control and Prevention (CDC)
      Ten great public health achievements–United States, 2001-2010.
      in the ideal world R/R would largely be funded by public health, even if delivered within health systems or by practices. Public health departments do report receiving funding for centralized R/R from grants from the Centers for Disease Control and Prevention (CDC), contractual agreements with the CDC, or as part of the CDC's Sentinel Sites Program.
      • Fisher MP
      • Gurfinkel D
      • Szilagyi PG
      • et al.
      Supporting and sustaining centralized reminder/recall for immunizations: qualitative insights from stakeholders.
      They sometimes receive supplies for R/R or other assistance from pharmaceutical companies or regional and state coalitions promoting immunizations and, infrequently, receive funds directly from the state.
      • Fisher MP
      • Gurfinkel D
      • Szilagyi PG
      • et al.
      Supporting and sustaining centralized reminder/recall for immunizations: qualitative insights from stakeholders.
      However, these opportunities are variable and there is rarely sustained, dependable funding for public health departments to conduct centralized R/R activities. Of note, New York City's IIS, in addition to other state IISs, has taken on a more hybrid approach that allows health care providers to identify children from their practices who are in the registry who need vaccination, and use the registry to create recall lists, letters or even to send a text message from the IIS.

      Citywide Immunization Registry. New Text Messaging via the Online Registry to Recall Patients Due/Overdue Immunizations. New York City Department of Health and Mental Hygiene. Available at: https://www1.nyc.gov/assets/doh/downloads/pdf/cir/txt-messaging-guide.pdf. Accessed February 1, 2021.

      There is little information about sources of funding for R/R conducted by health delivery systems, although pay for performance measures by insurers or HEDIS performance measures may make R/R efforts worth the investment if up-to-date rates for relevant vaccines are key performance indicators.
      • Fisher MP
      • Gurfinkel D
      • Szilagyi PG
      • et al.
      Supporting and sustaining centralized reminder/recall for immunizations: qualitative insights from stakeholders.
      Individual clinical practices may also benefit from such measures, but the lack of sustainment of practice-based R/R suggests this does not provide sufficient support for practices to maintain R/R.
      It seems logical that sustaining R/R efforts would optimally be supported via partnerships between the public and private entities that have a stake in raising immunization rates including health departments, insurers, health care systems and practices. Such collaborations would be ideal examples of the types of multi-sector initiatives that have been strongly supported by multiple reports by the Institutes of Medicine, now the National Academy of Medicine,
      Institute of Medicine Committee on Assuring the Health of the Public in the 21st Century
      The Future of the Public's Health in the 21st Century.
      Committee on Integrating Primary Care and Public Health, Board on Population Health and Public Health Practice, Institute of Medicine
      Primary Care and Public Health: Exploring Integration to Improve Population Health.
      Roundtable on Population Health Improvement, Board on Population Health and Public Health Practice, Institute of Medicine
      Collaboration Between Health Care and Public Health: Workshop Summary.
      the Robert Wood Johnson Foundation
      • Mattessich P
      • Rausch E.
      Collaboration to Build Healthier Communities: A Report for the Robert Wood Johnson Foundation Commission to Build a Healthier America.
      and other organizations advocating for collaborative models. They would have the potential to generate a larger collective impact and population-level impact than any one organization independently and, because costs would be shared, would be less of a burden on any single entity.

      Conclusions

      Despite some challenges posed by parental hesitancy, data integrity issues, and problems with sustainability, issues that are common to most strategies used to increase immunizations, evidence suggests that R/R is one of the most effective and cost-effective modalities and should continue to play a major role in efforts to increase immunization rates for many vaccines. Multi-level interventions, combining R/R with other interventions at the provider, practice, and system levels, will likely be needed to increase rates for those vaccines that are associated with higher levels of vaccine hesitancy. In addition, enhancement of R/R with educational or motivational messages and increasing the interactivity of messaging are innovations that may increase effectiveness in the face of vaccine hesitancy. Increased interoperability of electronic medical records and IISs over time with the potential for automatic updates of addresses from administrative data is likely to increase effectiveness and efficiency of centralized R/R using either IIS or other health system level data. Sustainment of R/R could optimally be supported via partnerships between public and private entities that have a stake in raising immunization rates including businesses, health departments, insurers, health care systems and practices. There are many examples of such cross-sector collaborations in community efforts to improve health which could serve as examples for R/R efforts
      Sciences Engineering Medicine
      Roundtable on Population Health Improvement.
      for an entire community. Such a collaborative approach would decrease the burden on each individual entity and increase the opportunity for a truly population-based approach to increasing immunization rates.

      Acknowledgments

      Financial disclosure: This article was published as part of a supplement sponsored by the Centers for Disease Control and Prevention.
      Disclaimer: The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

      References

        • Jacobson Vann JC
        • Jacobson RM
        • Coyne-Beasley T
        • et al.
        Patient reminder and recall interventions to improve immunization rates.
        Cochrane Database Syst Rev. 2018; 1CD003941https://doi.org/10.1002/14651858.CD003941.pub3
        • Jacobson Vann JC
        • Szilagyi P
        Patient reminder and patient recall systems to improve immunization rates.
        Cochrane Database Syst Rev. 2005; 2005Cd003941https://doi.org/10.1002/14651858.CD003941.pub2
        • Szilagyi PG
        • Bordley C
        • Vann JC
        • et al.
        Effect of patient reminder/recall interventions on immunization rates: a review.
        JAMA. 2000; 284: 1820-1827https://doi.org/10.1001/jama.284.14.1820
        • Szilagyi P
        • Vann J
        • Bordley C
        • et al.
        Interventions aimed at improving immunization rates.
        Cochrane Database Syst Rev. 2002; Cd003941https://doi.org/10.1002/14651858.cd003941
      1. Community Preventive Services Task Force. The Community Guide: Vaccination Programs: Client Reminder and Recall Systems. Centers for Disease Control and Prevention. Available at:https://www.thecommunityguide.org/findings/vaccination-programs-client-reminder-and-recall-systems. Accessed October 21, 2019.

        • Stockwell MS
        • Kharbanda EO
        • Martinez RA
        • et al.
        Text4Health: impact of text message reminder–recalls for pediatric and adolescent immunizations.
        Am J Public Health. 2012; 102: e15-e21https://doi.org/10.2105/ajph.2011.300331
        • Hofstetter AM
        • DuRivage N
        • Vargas CY
        • et al.
        Text message reminders for timely routine MMR vaccination: a randomized controlled trial.
        Vaccine. 2015; 33: 5741-5746https://doi.org/10.1016/j.vaccine.2015.09.042
        • O'Leary ST
        • Lee M
        • Lockhart S
        • et al.
        Effectiveness and cost of bidirectional text messaging for adolescent vaccines and well care.
        Pediatrics. 2015; 136: e1220-e1227https://doi.org/10.1542/peds.2015-1089
        • Mekonnen ZA
        • Gelaye KA
        • Were MC
        • et al.
        Effect of mobile text message reminders on routine childhood vaccination: a systematic review and meta-analysis.
        Syst Rev. 2019; 8: 154https://doi.org/10.1186/s13643-019-1054-0
        • Richman AR
        • Maddy L
        • Torres E
        • et al.
        A randomized intervention study to evaluate whether electronic messaging can increase human papillomavirus vaccine completion and knowledge among college students.
        J Am Coll Health. 2016; 64: 269-278https://doi.org/10.1080/07448481.2015.1117466
        • Patel A
        • Stern L
        • Unger Z
        • et al.
        Staying on track: a cluster randomized controlled trial of automated reminders aimed at increasing human papillomavirus vaccine completion.
        Vaccine. 2014; 32: 2428-2433https://doi.org/10.1016/j.vaccine.2014.02.095
        • Berenson AB
        • Rahman M
        • Hirth JM
        • et al.
        A human papillomavirus vaccination program for low-income postpartum women.
        Am J Obstet Gynecol. 2016; 215: 318.e1-318.e9https://doi.org/10.1016/j.ajog.2016.02.032
        • LeBaron CW
        • Starnes DM
        • Rask KJ.
        The impact of reminder-recall interventions on low vaccination coverage in an inner-city population.
        Arch Pediatr Adolesc Med. 2004; 158: 255-261https://doi.org/10.1001/archpedi.158.3.255
        • Hambidge SJ
        • Davidson AJ
        • Phibbs SL
        • et al.
        Strategies to improve immunization rates and well-child care in a disadvantaged population: a cluster randomized controlled trial.
        Arch Pediatr Adolesc Med. 2004; 158: 162-169https://doi.org/10.1001/archpedi.158.2.162
        • Kempe A
        • Lowery NE
        • Pearson KA
        • et al.
        Immunization recall: effectiveness and barriers to success in an urban teaching clinic.
        J Pediatr. 2001; 139: 630-635https://doi.org/10.1067/mpd.2001.117069
        • Irigoyen MM
        • Findley S
        • Wang D
        • et al.
        Challenges and successes of immunization registry reminders at inner-city practices.
        Ambul Pediatr. 2006; 6: 100-104https://doi.org/10.1016/j.ambp.2005.10.006
        • Rodewald LE
        • Szilagyi PG
        • Humiston SG
        • et al.
        A randomized study of tracking with outreach and provider prompting to improve immunization coverage and primary care.
        Pediatrics. 1999; 103: 31-38https://doi.org/10.1542/peds.103.1.31
        • Szilagyi PG
        • Humiston SG
        • Gallivan S
        • et al.
        Effectiveness of a citywide patient immunization navigator program on improving adolescent immunizations and preventive care visit rates.
        Arch Pediatr Adolesc Med. 2011; 165: 547-553https://doi.org/10.1001/archpediatrics.2011.73
        • Hambidge SJ
        • Phibbs SL
        • Chandramouli V
        • et al.
        A stepped intervention increases well-child care and immunization rates in a disadvantaged population.
        Pediatrics. 2009; 124: 455-464https://doi.org/10.1542/peds.2008-0446
        • Saville AW
        • Beaty B
        • Dickinson LM
        • et al.
        Novel immunization reminder/recall approaches: rural and urban differences in parent perceptions.
        Acad Pediatr. 2014; 14: 249-255https://doi.org/10.1016/j.acap.2014.02.003
        • Albright K
        • Hurley LP
        • Lockhart S
        • et al.
        Attitudes about adult vaccines and reminder/recall in a safety net population.
        Vaccine. 2017; 35: 7292-7296https://doi.org/10.1016/j.vaccine.2017.11.001
        • Hofstetter AM
        • Vargas CY
        • Kennedy A
        • et al.
        Parental and provider preferences and concerns regarding text message reminder/recall for early childhood vaccinations.
        Prev Med. Aug 2013; 57: 75-80https://doi.org/10.1016/j.ypmed.2013.04.007
        • Clark SJ
        • Butchart A
        • Kennedy A
        • et al.
        Parents' experiences with and preferences for immunization reminder/recall technologies.
        Pediatrics. 2011; 128: e1100-e1105https://doi.org/10.1542/peds.2011-0270
        • Kempe A
        • O'Leary ST
        • Shoup JA
        • et al.
        Parental choice of recall method for HPV vaccination: a pragmatic trial.
        Pediatrics. 2016; 137e20152857https://doi.org/10.1542/peds.2015-2857
        • Kempe A
        • Daley MF
        • Barrow J
        • et al.
        Implementation of universal influenza immunization recommendations for healthy young children: results of a randomized, controlled trial with registry-based recall.
        Pediatrics. 2005; 115: 146-154https://doi.org/10.1542/peds.2004-1804
        • Stockwell MS
        • Kharbanda EO
        • Martinez RA
        • et al.
        Effect of a text messaging intervention on influenza vaccination in an urban, low-income pediatric and adolescent population: a randomized controlled trial.
        JAMA. 2012; 307: 1702-1708
        • Tierney CD
        • Yusuf H
        • McMahon SR
        • et al.
        Adoption of reminder and recall messages for immunizations by pediatricians and public health clinics.
        Pediatrics. 2003; 112: 1076-1082https://doi.org/10.1542/peds.112.5.1076
        • Saville AW
        • Albright K
        • Nowels C
        • et al.
        Getting under the hood: exploring issues that affect provider-based recall using an immunization information system.
        Acad Pediatr. 2011; 11: 44-49
        • Kempe A
        • Saville A
        • Dickinson LM
        • et al.
        Population-based versus practice-based recall for childhood immunizations: a randomized controlled comparative effectiveness trial.
        Am J Public Health. 2013; 103: 1116-1123https://doi.org/10.2105/AJPH.2012.301035
        • Kempe A
        • Saville AW
        • Dickinson LM
        • et al.
        Collaborative centralized reminder/recall notification to increase immunization rates among young children: a comparative effectiveness trial.
        JAMA Pediatr. 2015; 169: 365-373https://doi.org/10.1001/jamapediatrics.2014.3670
        • Saville AW
        • Gurfinkel D
        • Beaty BL
        • et al.
        The potential for centralized reminder/recall to increase immunization rates: a national survey of immunization information systems (IIS) managers.
        Prev Med Rep. 2021; 21101296https://doi.org/10.1016/j.pmedr.2020.101296
        • Szilagyi PG
        • Albertin C
        • Humiston SG
        • et al.
        A randomized trial of the effect of centralized reminder/recall on immunizations and preventive care visits for adolescents.
        Acad Pediatr. 2013; 13: 204-213https://doi.org/10.1016/j.acap.2013.01.002
        • Rand CM
        • Brill H
        • Albertin C
        • et al.
        Effectiveness of centralized text message reminders on human papillomavirus immunization coverage for publicly insured adolescents.
        J Adolesc Health. 2015; 56: S17-S20https://doi.org/10.1016/j.jadohealth.2014.10.273
        • Dombkowski KJ
        • Harrington LB
        • Dong S
        • et al.
        Seasonal influenza vaccination reminders for children with high-risk conditions: a registry-based randomized trial.
        Am J Prev Med. 2012; 42: 71-75
        • Chao C
        • Preciado M
        • Slezak J
        • et al.
        A randomized intervention of reminder letter for human papillomavirus vaccine series completion.
        J Adolesc Health. 2015; 56: 85-90https://doi.org/10.1016/j.jadohealth.2014.08.014
        • Kempe A
        • Saville AW
        • Albertin C
        • et al.
        Centralized reminder/recall to increase influenza vaccination rates: a two-state pragmatic randomized trial.
        Acad Pediatr. 2020; 20: 374-383https://doi.org/10.1016/j.acap.2019.10.015
        • Szilagyi PG
        • Albertin CS
        • Saville AW
        • et al.
        Effect of state immunization information system based reminder/recall for influenza vaccinations: a randomized trial of autodialer, text, and mailed messages.
        J Pediatr. 2020; 221: 123-131.e4https://doi.org/10.1016/j.jpeds.2020.02.020
        • Szilagyi P
        • Albertin C
        • Gurfinkel D
        • et al.
        Effect of state immunization information system centralized reminder and recall on HPV vaccination rates.
        Pediatrics. 2020; 145https://doi.org/10.1542/peds.2019-2689
      2. Gurfinkel D, Kempe A, Albertin C, et al. Centralized reminder/recall of HPV vaccination: findings from two states: a randomized clinical trial. J Adolesc Health. In press.

        • Kempe A
        • Saville AW
        • Albertin C
        • et al.
        Parental hesitancy about routine childhood and influenza vaccinations: a national survey.
        Pediatrics. 2020; 146https://doi.org/10.1542/peds.2019-3852
        • Szilagyi PG
        • Albertin CS
        • Gurfinkel D
        • et al.
        Prevalence and characteristics of HPV vaccine hesitancy among parents of adolescents across the US.
        Vaccine. 2020; 38: 6027-6037https://doi.org/10.1016/j.vaccine.2020.06.074
        • Hofstetter AM
        • Vargas CY
        • Camargo S
        • et al.
        Impacting delayed pediatric influenza vaccination: a randomized controlled trial of text message reminders.
        Am J Prev Med. 2015; 48: 392-401https://doi.org/10.1016/j.amepre.2014.10.023
        • Jenssen BP
        • Buttenheim AM
        • Fiks AG.
        Using behavioral economics to encourage parent behavior change: opportunities to improve clinical effectiveness.
        Acad Pediatr. 2019; 19: 4-10https://doi.org/10.1016/j.acap.2018.08.010
      3. Lerner C, Albertin C, Casillas A, et al. Patient portal reminders for pediatric influenza vaccinations: a randomized clinical trial. Pediatrics. In press.

        • Stockwell MS
        • Hofstetter AM
        • DuRivage N
        • et al.
        Text message reminders for second dose of influenza vaccine: a randomized controlled trial.
        Pediatrics. 2015; 135: e83-e91https://doi.org/10.1542/peds.2014-2475
        • Hughes MM
        • Reed C
        • Flannery B
        • et al.
        Projected population benefit of increased effectiveness and coverage of influenza vaccination on influenza burden in the United States.
        Clin Infect Dis. 2020; 70: 2496-2502https://doi.org/10.1093/cid/ciz676
        • Yusuf H
        • Adams M
        • Rodewald L
        • et al.
        Fragmentation of immunization history among providers and parents of children in selected underserved areas.
        Am J Prev Med. 2002; 23: 106-112https://doi.org/10.1016/s0749-3797(02)00463-4
        • Kolasa MS
        • Chilkatowsky AP
        • Clarke KR
        • et al.
        How complete are immunization registries? The Philadelphia story.
        Ambul Pediatr. 2006; 6: 21-24https://doi.org/10.1016/j.ambp.2005.08.006
        • Bardenheier BH
        • Yusuf HR
        • Rosenthal J
        • et al.
        Factors associated with underimmunization at 3 months of age in four medically underserved areas.
        Public Health Rep. 2004; 119: 479-485https://doi.org/10.1016/j.phr.2004.07.005
        • Kempe A
        • Saville AW
        • Beaty B
        • et al.
        Centralized reminder/recall to increase immunization rates in young children: how much bang for the buck?.
        Acad Pediatr. 2017; 17: 330-338https://doi.org/10.1016/j.acap.2016.11.016
        • Jacob V
        • Chattopadhyay SK
        • Hopkins DP
        • et al.
        Increasing coverage of appropriate vaccinations: a community guide systematic economic review.
        Am J Prev Med. 2016; 50: 797-808https://doi.org/10.1016/j.amepre.2015.11.003
        • Centers for Disease Control and Prevention (CDC)
        Ten great public health achievements–United States, 2001-2010.
        MMWR Morb Mortal Wkly Rep. 2011; 60: 619-623
        • Fisher MP
        • Gurfinkel D
        • Szilagyi PG
        • et al.
        Supporting and sustaining centralized reminder/recall for immunizations: qualitative insights from stakeholders.
        Vaccine. 2019; 37: 6601-6608https://doi.org/10.1016/j.vaccine.2019.09.055
      4. Citywide Immunization Registry. New Text Messaging via the Online Registry to Recall Patients Due/Overdue Immunizations. New York City Department of Health and Mental Hygiene. Available at: https://www1.nyc.gov/assets/doh/downloads/pdf/cir/txt-messaging-guide.pdf. Accessed February 1, 2021.

        • Institute of Medicine Committee on Assuring the Health of the Public in the 21st Century
        The Future of the Public's Health in the 21st Century.
        National Academies Press (US), 2002
        • Committee on Integrating Primary Care and Public Health, Board on Population Health and Public Health Practice, Institute of Medicine
        Primary Care and Public Health: Exploring Integration to Improve Population Health.
        National Academies Press (US), 2012
        • Roundtable on Population Health Improvement, Board on Population Health and Public Health Practice, Institute of Medicine
        Collaboration Between Health Care and Public Health: Workshop Summary.
        National Academies Press (US), 2016
        • Mattessich P
        • Rausch E.
        Collaboration to Build Healthier Communities: A Report for the Robert Wood Johnson Foundation Commission to Build a Healthier America.
        2013: 24
        • Sciences Engineering Medicine
        Roundtable on Population Health Improvement.
        National Academies of Sciences, 2021 (Available at:) (Accessed February 24, 2021)