Advertisement

Rapid Development and Testing of a COVID-19 Vaccine Curriculum for Pediatricians

Published:September 16, 2022DOI:https://doi.org/10.1016/j.acap.2022.09.010

      Abstract

      Background and Objectives: As the coronavirus disease 2019 (COVID-19) pandemic evolves and vaccines become available to children, pediatricians must navigate vaccination discussions in the setting of rapidly changing vaccine recommendations and approvals. We developed and evaluated an educational curriculum for pediatricians to improve their knowledge about COVID-19 vaccines and confidence in communicating with patients and families about COVID-19 vaccines.
      Methods: Five institutions collaborated to develop an online educational curriculum. Utilizing the collaboration's multidisciplinary expertise, we developed a three-module curriculum focused on the SARS-CoV-2 virus and vaccine basics, logistics and administration of COVID-19 vaccine, and COVID-19 vaccine communication principles. Surveys administered to clinician participants before and after completion of the curriculum assessed knowledge and confidence; a follow-up survey 1 month after the post-survey assessed persistence of initial findings.
      Results: A total of 152 pediatric providers participated; 72 completed both pre- and post-surveys. The median knowledge score improved from the pre-survey to the post-survey (79% to 93%, P < 0.001). There was an increase in providers’ confidence after completing the curriculum, which persisted in the follow-up survey. In the post-survey, 98% of participants had had the opportunity to discuss the COVID-19 vaccine with patients, and most clinicians reported that the modules decreased apprehension some or significantly.
      Conclusions: This project demonstrates rapid and feasible deployment of a curriculum providing up-to-date information to front-line clinicians responsible for having complex conversations about COVID-19 vaccine decision-making. Clinicians who completed this curriculum had sustained increased confidence and decreased levels of apprehension when discussing the COVID-19 vaccine.

      Keywords

      Introduction

      As the coronavirus disease 2019 (COVID-19) pandemic has evolved, children have accounted for a large proportion of COVID-19 infections. Early on, children represented 2% to 10% of infections
      • Posfay-Barbe KM
      • Wagner N
      • Gauthey M
      • et al.
      COVID-19 in children and the dynamics of infection in families.
      ; however, as mutations in the SARS-CoV-2 virus caused it to be more contagious, there was limited vaccination among children.
      • Wanga V
      • Gerdes ME
      • Shi DS
      • et al.
      Characteristics and clinical outcomes of children and adolescents aged <18 years hospitalized with COVID-19—six hospitals, United States, July–August 2021.
      As the Delta variant became the predominant strain circulating in the US, increased transmission in households from child to adult was noted.
      • Chu VT
      • Yousaf AR
      • Chang K
      • et al.
      Georgia Camp Investigation Team. Household transmission of SARS-CoV-2 from children and adolescents.
      ,
      • Paul LA
      • Daneman N
      • Schwartz KL
      • et al.
      Association of age and pediatric household transmission of SARS-CoV-2 infection.
      As Delta became the predominant variant over the summer and early fall months of 2021, the percentage of cases in children rose to represent a quarter of new infections.

      American Academy of Pediatrics. Children and COVID-19: State-level data report. Accessed November 2, 2021. https://www.aap.org/en/pages/2019-novel-coronavirus-covid-19-infections/children-and-covid-19-state-level-data-report. Updated 2021.

      A COVID-19 vaccine became available for adolescents ≥12 years old in May 2021 and for children ≥5 years old in November 2021, placing pediatricians at the forefront of vaccine promotion.

      US Food and Drug Administration. Coronavirus (COVID-19) update: FDA authorizes Pfizer-BioNTech COVID-19 vaccine for emergency use in adolescents in another important action in fight against pandemic. Accessed May 10, 2021. https://www.fda.gov/news-events/press-announcements/coronavirus-covid-19-update-fda-authorizes-pfizer-biontech-covid-19-vaccine-emergency-use.

      ,

      US Food and Drug Administration. FDA authorizes Pfizer-BioNTech COVID-19 vaccine for emergency use in children 5 through 11 years of age. Accessed October 29, 2021. https://www.fda.gov/news-events/press-announcements/fda-authorizes-pfizer-biontech-covid-19-vaccine-emergency-use-children-5-through-11-years-age.

      Discussions with parents about COVID-19 vaccines for children, however, is largely uncharted territory given a number of unique factors. First, the Pfizer and Moderna vaccines were the first publicly available vaccines to use messenger RNA technology.
      Centers for Disease Control and Prevention
      Understanding mRNA COVID-19 vaccines.
      Second, COVID-19 vaccines were the first vaccines made available to the broader public through the US Food and Drug Administration's emergency use authorization mechanism. Third, the infectious disease transmission dynamics of the virus rapidly changed with each new variant throughout the course of the pandemic. Fourth, COVID-19 vaccine distribution and administration required specific packaging, storage, and handling requirements, as well as monitoring following vaccination.
      Centers for Disease Control and Prevention
      COVID-19 vaccination.
      For example, many practices did not have access to ultracold storage and thus felt ill equipped initially to provide the COVID-19 vaccine in their clinics. Others struggled with operationalization of a prolonged 15- to 30-minute observation period post-vaccination, while maintaining physical distance. All of these factors had the potential to limit clinicians’ knowledge of COVID-19 vaccines, limit their ability to provide vaccines in their practice, and negatively impact their confidence in discussing these vaccines with families.
      Centers for Disease Control and Prevention
      COVID-19 vaccination.
      For these reasons, accessible, accurate, and up-to-date information for clinicians was needed so they could feel equipped to answer parent questions on COVID-19 vaccines. There was also a critical need to create a timely COVID-19 vaccine–specific curriculum for pediatric clinicians. Clinicians are a trusted source of information––the most trusted source of COVID-19 vaccine information for parents

      Hamel L, Lopes L, Kearney A, et al. KFF COVID-19 vaccine monitor: Winter 2021 update on parents’ views of vaccines for kids. Kaiser Family Foundation. December 9, 2021. https://www.kff.org/coronavirus-covid-19/poll-finding/kff-covid-19-vaccine-monitor-winter-2021-update-on-parents-views-of-vaccines/

      ––and by providing a confident vaccine recommendation combined with an understanding of specific parental concerns and empathetic storytelling, clinicians play a pivotal role in empowering parental vaccine decision-making.
      • Marotta S
      • McNally VV.
      Increasing vaccine confidence through parent education and empowerment using clear and comprehensible communication.
      For instance, a strong recommendation from a pediatrician has been shown to be the single most influential factor for vaccine acceptance.
      • Freed GL
      • Clark SJ
      • Butchart AT
      • Singer DC
      • Davis MM.
      Sources and perceived credibility of vaccine-safety information for parents.
      ,
      • Szilagyi PG
      • Shah MD
      • Delgado JR
      • et al.
      Parents’ intentions and perceptions about COVID-19 vaccination for their children: results from a national survey.
      While the Centers for Disease Control and Prevention (CDC) released COVID-19 vaccine curricula in December 2020, the content was general and the modules were strictly text-based.

      Centers for Disease Control and Prevention. COVID-19 Vaccine Training Modules. https://www2.cdc.gov/vaccines/ed/covid19/. Accessed April 4, 2022.

      The objective of this study was to rapidly develop and test an interactive, online COVID-19 vaccine curriculum for pediatric providers.

      Methods

      Conceptual Framework

      In developing a systematic approach for this project, the core concepts we considered were disseminating COVID-19 vaccine research and helping clinicians communicate the importance of vaccination to their patients and parents. Wilson et al summarized various frameworks to address dissemination of new research.
      • Wilson PM
      • Petticrew M
      • Calnan MW
      • Nazareth I.
      Disseminating research findings: what should researchers do? A systematic scoping review of conceptual frameworks.
      However, most reviewed frameworks targeted researchers or nonphysician audiences or focused on culture and quality.
      • Wilson PM
      • Petticrew M
      • Calnan MW
      • Nazareth I.
      Disseminating research findings: what should researchers do? A systematic scoping review of conceptual frameworks.
      We ultimately chose to anchor our approach on Winkler's framework for disseminating new concepts to practicing physicians,
      • Winkler JD
      • Lohr KN
      • Brook RH.
      Persuasive communication and medical technology assessment.
      Lasswell's foundational concept of “who says what in which channel to whom with what effect,”
      • Lasswell HD.
      The structure and function of communication in society.
      and McGuire's concept of input and output variables in his communication and persuasion matrix.

      McGuire W. McGuire's classic input-output framework for constructing persuasive messages. In: Rice RE, Atkin CK, eds. Public Communication Campaigns (4th ed.). Washington, DC: Sage; 2013: 133–145.

      Applying this framework, a collaborative of pediatric academic institutions delivered COVID-19 vaccine education through interactive online modules to general pediatric clinicians with an intent to improve knowledge and confidence.

      Curriculum Design

      Kern's curriculum design process was applied in a step-wise fashion in developing and implementing this learning initiative.
      • Thomas PA
      • Kern DE
      • Hughes MT
      • Chen BY
      Curriculum Development for Medical Education: A Six-Step Approach.
      Given the continuing effects of the pandemic, some steps were completed rapidly or in parallel with other steps.

      Step 1. General Needs Assessment

      A collaborative was formed in January 2021 among five institutions: Children's Hospital of Alabama, Children's Mercy, Children's National, Lurie Children's, and Seattle Children's. Experts in general pediatrics, pediatric infectious diseases, medical education, simulation, and disaster preparedness gathered to identify key areas of focus in the rapidly evolving COVID-19 pandemic. Discussions with families about COVID-19 vaccines needed to begin early in anticipation that vaccination of children would be indicated as vaccine supply increased, high-risk groups were vaccinated, and pediatric data were obtained. A review of literature related to vaccine education identified what was known and gaps to fill through this educational initiative.
      • Opel DJ
      • Lo B
      • Peek ME.
      Addressing mistrust about COVID-19 vaccines among patients of color.
      • Koski K
      • Lehto JT
      • Hakkarainen K.
      Simulated encounters with vaccine-hesitant parents: arts-based video scenario and a writing exercise.
      • Nold L
      • Deem MJ.
      A simulation experience for preparing nurses to address refusal of childhood vaccines.
      • Oduwole EO
      • Pienaar ED
      • Mahomed H
      • Wiysonge CS.
      Current tools available for investigating vaccine hesitancy: a scoping review protocol.
      • Pahud B
      • Williams SE
      • Lee BR
      • et al.
      A randomized controlled trial of an online immunization curriculum.
      • Real FJ
      • DeBlasio D
      • Beck AF
      • et al.
      A virtual reality curriculum for pediatric residents decreases rates of influenza vaccine refusal.
      • Shalansky RA
      • Wu M
      • Shen SC
      • et al.
      Evaluation of a pilot immunization curriculum to meet competency training needs of medical residents.

      Step 2. Targeted Needs Assessment

      With the rapidly evolving knowledge in the scientific community around COVID-19 vaccines, the group decided to target education to pediatric clinicians. Engaging the target audience––general pediatricians in academic and community practice––in the planning group and curriculum development identified specific areas of need.

      Step 3. Developing Goals and Objectives

      The goal of the curriculum was to educate pediatric clinicians about COVID-19 vaccine technology; describe how a clinic could handle logistics, storage, and administration of COVID-19 vaccines; and allow clinicians to practice communication strategies that could promote vaccine acceptance. To achieve this goal, we created three modules. Module 1 covered the SARS-CoV-2 virus and vaccine basics, Module 2 focused on how to implement a COVID-19 vaccine clinic, and Module 3 focused on COVID-19 vaccine communication strategies to use in the vaccine encounter. The modular objectives are listed in Table 1.
      Table 1COVID-19 Vaccine Education Curriculum
      Module titlePrimary objectiveKey concepts
      Module 1:

      Basics of COVID-19 Vaccines
      Describe the SARS-CoV-2 virus and its variants.

      Explain the COVID-19 vaccines and CDC recommendations.
      Nomenclature, definitions, and facts

      Rise of SARS-COV-2 virus variants

      Vaccine effectiveness

      Emergency use authorization
      Module 2:

      COVID-19 Vaccine Logistics
      Demonstrate best practices in COVID-19 vaccine distribution, storage, administration, record keeping, and reimbursementVaccine logistics

      Available COVID-19 vaccines in the United States

      Enrolling as a vaccine provider

      Vaccine storage and handling

      Simulation guide for a COVID-19 vaccine clinic
      Module 3:

      Addressing Vaccine Acceptance
      Demonstrate best practices in addressing COVID-19 vaccine hesitancyKey principles in promoting vaccine acceptance

      Opportunity to practice conversations with two simulated patient cases

      Recommended response with highlighted key principles and motivational interviewing techniques

      Conversation on COVID-19 vaccine equity

      Step 4. Educational Strategies

      As the collaborative formed, many ideas related to the curriculum emerged and the members refined the content to ensure relevance to the goals and the audience. With rapidly emerging literature and information about COVID-19 and its vaccines, the authors had to select and prioritize the most relevant curriculum content to deliver to the target audience of pediatric clinicians.
      Module 1 covered nomenclature, definitions, and facts surrounding the SARS-CoV-2 virus and the COVID-19 vaccines. This introductory module began by distinguishing the nomenclature of SARS-CoV-2 and the disease COVID-19 followed by a discussion of transmissibility through understanding of the basic reproduction number (R0) and comparison to other diseases such as measles, polio, and influenza. It also included a timeline
      World Health Organization
      Timeline: WHO's COVID-19 response.
      of the COVID-19 pandemic from first case detection to the creation of the first vaccines.

      US Food and Drug Administration. Commissioner of the FDA takes key action in fight against COVID-19 by issuing emergency use authorization for first COVID-19 vaccine. December 11, 2020. https://www.fda.gov/news-events/press-announcements/fda-takes-key-action-fight-against-covid-19-issuing-emergency-use-authorization-first-covid-19.

      The next part focused on the development of the COVID-19 vaccines, including the role of targeting spike proteins,

      National Foundation for Infectious Diseases. Coronaviruses. July 27, 2021. https://www.nfid.org/infectious-diseases/coronaviruses/.

      the viral genome, and the four types of COVID-19 vaccines. The final part of Module 1 discussed the rise of SARS-CoV-2 virus variants,
      Centers for Disease Control and Prevention
      SARS-COV-2 variant classifications and definitions.
      vaccine effectiveness,

      Woodward A. One chart shows how well COVID-19 vaccines work against the 3 most worrisome coronavirus variants. Business Insider. March 12, 2021. https://www.businessinsider.com/covid-19-vaccines-efficacy-variants-south-africa-uk-brazil-2021-3

      and emergency use authorization.

      Center for Biologics Evaluation and Research. Emergency use authorization for vaccines explained. Accessed October 13, 2021. https://www.fda.gov/vaccines-blood-biologics/vaccines/emergency-use-authorization-vaccines-explained.

      Module 2 covered vaccine logistics and best practices for vaccine distribution, storage, administration, record keeping, and reimbursement. Learners reviewed the COVID-19 vaccines available in the United States and were led through the process of enrolling as a vaccine provider, highlighting training from the CDC

      Centers for Disease Control and Prevention. COVID-19 Vaccine Training Modules. https://www2.cdc.gov/vaccines/ed/covid19/. Accessed April 4, 2022.

      and eligibility requirements.

      US Department of Health and Human Services. Register today! Become a part of the expanded COVID-19 vaccine workforce. Accessed October 13, 2021. https://www.phe.gov/emergency/events/COVID19/COVIDvaccinators/Pages/register.aspx.

      Information on vaccine storage and handling from the CDC

      Centers for Disease Control and Prevention. Administration overview for Pfizer-Biontech COVID-19 vaccine. August 25, 2021. https://www.cdc.gov/vaccines/covid-19/info-by-product/pfizer/index.html.

      was compiled into a table, and guidance was reviewed on administration, precautions and contraindications,
      Centers for Disease Control and Prevention
      Appendix B: Triage of people with a history of allergies or allergic reactions.
      and post-vaccination monitoring. Tools for billing and coding were provided through the Centers for Medicare and Medicaid Services, American Medical Association, and American Academy of Pediatrics. The module concluded with a guide to simulate a COVID-19 vaccine clinic to maximize efficiency and safety. This guide instructed clinicians in (1) completing the pre-work of logistics, personnel, and contingencies; (2) setting up the simulation with timing, flow, and debriefing; (3) running the simulation, including orientation and execution; and (4) debriefing, with a focus on space, structure, and gathering ideas to successfully execute a COVID-19 vaccine clinic.
      Module 3 covered key general principles in promoting vaccine acceptance based on a review of the literature. These principles included leading with listening, tailoring responses to a patient/parent's specific concern, acknowledging uncertainty, using accessible language, not repeating myths, and recommending with confidence.
      • Opel DJ
      • Lo B
      • Peek ME.
      Addressing mistrust about COVID-19 vaccines among patients of color.
      ,
      • Bryant KA
      • Wesley GC
      • Wood JA
      • Hines C
      • Marshall GS.
      Use of standardized patients to examine physicians’ communication strategies when addressing vaccine refusal: a pilot study.
      • Loehr J
      • Savoy M.
      Strategies for addressing and overcoming vaccine hesitancy.

      Olson O, Berry C, Kumar N. Addressing parental vaccine hesitancy towards childhood vaccines in the United States: a systematic literature review of communication interventions and strategies. Vaccines (Basel). 2020;8(4):590. doi:10.3390/vaccines8040590.

      The module provided a summary of motivational interviewing techniques
      • Opel DJ
      • Lo B
      • Peek ME.
      Addressing mistrust about COVID-19 vaccines among patients of color.
      and the opportunity to practice conversations through two simulated patient cases involving an African American boy with severe asthma and a pregnant mother with an immunocompromised child. The team carefully selected cases to address known racial disparities related to COVID-19, allergic reaction concerns associated with the vaccines, and hesitancy in vaccination during pregnancy and in those who are immune compromised. A brief video presented a parent posing questions. Learners recorded their response via text, audio, or video and viewed a video of a recommended response that highlighted key principles and motivational interviewing techniques. Finally, learners listened to a conversation in podcast format on COVID-19 vaccine equity with Nathan Chomilo, a general pediatrician, medical director for the State of Minnesota's Medicaid and MinnesotaCare programs, and Minnesota's director for vaccine equity.
      To reach the target audience of pediatric clinicians during the COVID-19 pandemic when in-person, synchronous instruction was limited, we selected an educational strategy that involved computer-based instruction and interaction. We delivered three interactive modules with quizzes, videos, and links spaced throughout the curricular content using the Articulate 360 e-learning suite (articulate.com/360) and a Moodle-based learning management system (moodle.org). Learners set up a login and password to access the learning management system and completed the content at their own pace. The content was developed and refined between February and April 2021. The curriculum and modules are available at https://www.childrensmedicaleducation.org/nppcc/login/index.php.

      Step 5. Implementation

      As data were collected to demonstrate the benefit of the modules, each individual site worked with its respective institutional review board (IRB). At all five sites, the IRBs deemed this study to be exempt, as the research was to review the anonymized results of educational tests. Each potential learner received an IRB-approved information sheet describing the research study.
      Members of the collaborative piloted the modules in April 2021 and provided valuable feedback to refine the content and delivery. Internal grants from each of the five collaborative members provided funding for this initiative. The five institutions connected with primary care pediatrics leadership in their local hospitals, affiliated practices, and local pediatric associations to distribute this educational opportunity. Between May and September 2021, providers were recruited by sending an invitation to participate via email to pediatric associations and affiliated practice listservs as well as giving presentations to those practices and associations about the study.
      Upon completion of the curriculum, three Continuing Medical Education credits and three American Board of Pediatrics Maintenance of Certification Part 2 points were awarded to each physician. If a COVID-19 vaccine clinic simulation was performed, physicians were able to earn up to two additional American Board of Pediatrics Maintenance of Certification Part 2 points. For advanced practice nurses, 3 hours of Continuing Nurse Education credits were provided. As the pandemic evolved, new guidelines and knowledge regarding COVID-19 illness and vaccines were incorporated into Modules 1 and 2 and delivered to new participants. Every other week, the team met and executed modifications to promote enrollment, including changing the order of Modules 2 and 3, making the continuing education credit information available upon signing up, and emphasizing the option to submit a text response rather than require an audio or video recording in Module 3 to promote completion of all the modules.

      Step 6. Evaluation and Feedback

      To demonstrate the effectiveness of this initiative, we administered online surveys to participants before completion of the modules (pre-survey), immediately after completion (post-survey), and 1 month after completion of the post-survey (follow-up survey) (Supplemental Material). The pre- and post-surveys assessed participants’ knowledge about SARS-CoV-2 virus and COVID-19 vaccines with a 14-question knowledge evaluation tool blueprinted to the objectives and content. The pre- and post-surveys also assessed confidence with 11 questions about self-efficacy adapted from a previous study that utilized a 5-point Likert response (1, “not at all confident”; 2, “slightly confident”; 3, “somewhat confident”; 4, “mostly confident”; and 5, “very confident”).
      • Henrikson NB
      • Opel DJ
      • Grothaus L
      • et al.
      Physician communication training and parental vaccine hesitancy: a randomized trial.
      The pre-survey also included demographic questions regarding clinicians’ highest level of education (physician, advanced practice provider), years in practice, practice setting (urban, suburban, rural, multiple settings), and proportion of patients in their practice panel who receive Medicaid and who are Black, Indigenous, or a person of color. We obtained this information to ensure we reached a broad, diverse population of pediatric clinicians. The post-survey requested feedback on the modules. The follow-up survey was created to assess persistence in confidence using selected confidence questions from the post-survey. It also included items prompting participants to describe opportunities in vaccine conversations and clinic planning. In an effort to minimize the time required to complete the follow-up survey, we opted not to conduct a repeat knowledge assessment.
      Applying Messick's framework for validity, to assess content and the response process, all surveys were piloted through members of the study team and other partners in the collaborative, including educational experts and primary care providers, to ensure clear understanding of the questions and responses.
      • Cook DA
      • Beckman TJ.
      Current concepts in validity and reliability for psychometric instruments: theory and application.
      Time and the small number of responses limited the ability to obtain further validity evidence on the survey instruments, such as internal structure, relation to other variables, and consequences.

      Data Analysis

      Our primary outcomes were knowledge score and perceived confidence. The proportion of correct responses on the 14 knowledge questions was calculated on the pre- and post-survey for each participant. The median response on the Likert scale for each of the 11 confidence questions was tabulated for both pre- and post-surveys. Comparisons between pre-survey and post-survey were restricted to participants who completed both surveys. Nonparametric summary distributions for both knowledge score and confidence were compared using the Wilcoxon signed-rank test. Effect sizes were calculated using matched-pairs rank biserial correlation, where a large effect size is noted at values >0.37.
      • Kerby DS.
      The simple difference formula: an approach to teaching nonparametric correlation.
      ,
      • Ruscio J.
      A probability-based measure of effect size: robustness to base rates and other factors.
      Four confidence questions that were assessed on the pre- and post-survey were also included on the follow-up survey administered 1 month after completion of the curriculum and post-survey. These responses were compared to the post-survey using the Wilcoxon signed-rank test to demonstrate sustainability. Surveys categorized by the learning management system as failed, incomplete, or not attempted were excluded.
      To assess for the presence of selection biases between those who completed both surveys and those who completed only the pre-survey, we compared the pre-survey responses between those who did versus those who did not complete the post-survey using Pearson's chi-square test for select categorical demographic characteristics and the Wilcoxon rank-sum test for continuous factors (i.e., years of practice, knowledge score, and confidence). All analyses were completed using R software (version 4.0.3; R Core Team, Vienna, Austria).

      Results

      A total of 152 pediatric providers from 11 US states participated in the interactive educational modules from May to September 2021. Participants had been in practice for a median of 14 years, and most were physicians (Table 2). Just over half of participants reported working in an urban setting. About one-third reported that >50% of their patients had Medicaid, and about one-quarter reported that >50% of their patients identified as Black, indigenous, or people of color.
      Table 2Participant Characteristics (n = 152)
      CategoryVariableN (%)
      StateAlabama11 (7%)
      California1 (1%)
      District of Columbia28 (18%)
      Illinois11 (7%)
      Indiana1 (1%)
      Kansas32 (21%)
      Maryland9 (6%)
      Missouri27 (18%)
      Texas1 (1%)
      Virginia5 (3%)
      Washington26 (17%)
      Practice settingUrban88 (58%)
      Suburban54 (36%)
      Rural4 (3%)
      Multiple settings3 (2%)
      Unknown3 (2%)
      Highest degree of educationPhysician130 (86%)
      Advanced practice provider22 (14%)
      Percent Medicaid0–25%43 (28%)
      26–50%28 (18%)
      51–75%15 (10%)
      76% or higher39 (26%)
      Don't know/unanswered27 (18%)
      Percent Black, indigenous, and people of color0–25%28 (18%)
      26–50%19 (13%)
      51–75%21 (14%)
      76% or higher25 (16%)
      Don't know/unanswered59 (39%)
      Years in practiceMedian [interquartile range]14 [7, 21]
      Among the 72 participants who completed both the pre-survey and post-survey, the median knowledge score was 79% on the pre-survey and 93% on the post-survey (P < 0.001) with an effect size of 0.95 [IQR: 0.86, 1.00] as measured by the matched-pairs rank biserial correlation (Figure 1). For each of the 11 confidence questions, there was a statistically significant increase in confidence in the post-survey with large effect sizes (Table 3). For three of the four confidence questions in the 1-month follow-up survey, the statistically significant increase in confidence persisted as well, except for discussing the risks of the vaccine (Table 4). There were no significant differences in participant characteristics and knowledge among those who did and did not complete both the pre- and post-surveys. For confidence in discussing the COVID-19 vaccines, most of the comparisons were also not significant (Supplemental Tables 1 and 2).
      Figure 1
      Figure 1Increase in knowledge about COVID-19 vaccines following the educational intervention (N = 72). This box and whiskers plot shows the median (solid bar), 25th to 75th percentiles (white box), 5th to 95th percentiles (vertical lines), and outliers (bullets).
      Table 3Change in Confidence in Discussing COVID-19 Vaccines
      How confident are you in your ability to…(N=72)Median [interquartile range]P

      value
      Based on Wilcoxon signed-rank test comparing pre-survey to post-survey.
      Effect size
      Based on matched-pairs rank biserial correlation.
      [IQR]
      Pre-surveyPost-survey
      Communicate with parents about COVID-19 vaccines4 [3.75, 4]5 [4, 5]<0.0010.93

      [0.81, 1.00]
      Establish an ongoing dialogue about COVID-19 vaccines4 [4, 5]5 [4, 5]<0.0010.87

      [0.66, 1.00]
      Provide COVID-19 vaccine information resources4 [3, 4]5 [4, 5]<0.0010.94

      [0.83, 1.00]
      Answer parent questions about COVID-19 vaccines4 [3, 4.25]5 [4, 5]<0.0010.92

      [0.79, 1.00]
      Address parent concerns about COVID-19 vaccines4 [3, 4]5 [4, 5]<0.0010.97

      [0.89, 1.00]
      Address specific concerns in Black, indigenous, and people of color populations4 [3, 4]4 [4, 5]<0.0010.94

      [0.84, 1.00]
      Talk about the benefits of COVID-19 vaccines for children4 [3, 5]5 [4, 5]<0.0010.92

      [0.78, 1.00]
      Talk about the risks of COVID-19 vaccines for children4 [3, 4.25]5 [4, 5]<0.0010.94

      [0.84, 1.00]
      Administer COVID-19 vaccine to your patients4 [3, 4.25]5 [4, 5]<0.0010.68

      [0.45, 0.86]
      Conduct a COVID-19 vaccine clinic4 [2.75, 4]4 [4, 5]<0.0010.70

      [0.48, 0.88]
      Prepare your practice team for vaccine delivery3 [3, 4]4 [4, 5]<0.0010.86

      [0.71, 0.97]
      low asterisk Based on Wilcoxon signed-rank test comparing pre-survey to post-survey.
      low asterisklow asterisk Based on matched-pairs rank biserial correlation.
      Table 4Change in Confidence in Discussing COVID-19 Vaccines on Follow-up Survey
      How confident are you in your ability to… (N=44)Median [interquartile range]P

      value*
      Pre-

      survey
      Post-

      survey
      Follow-up

      survey
      Communicate with parents about COVID-19 vaccines4 [3, 4]5 [4, 5]5 [4, 5]0.248
      Talk about the benefits of COVID-19 vaccines for children4 [3, 5]5 [4, 5]5 [4, 5]0.972
      Talk about the risks of COVID-19 vaccines for children4 [3, 4]5 [4, 5]4 [4, 5]0.012
      Administer COVID-19 vaccine to your patients4 [3, 4.5]4 [4, 5]4 [4, 5]0.605
      *Based on Wilcoxon signed-rank test comparing post-survey to follow-up survey.
      In feedback on the modules, 69 of 72 participants (96%) found the modules mostly or very helpful in providing education on COVID-19 vaccines and communicating with parents. In addition, 62 of 72 participants (86%) reported the modules as mostly or very helpful in developing a plan to administer the COVID-19 vaccine in their clinic (Figure 2). When asked to provide feedback about the modules, participants reported positive comments overall, indicating that the information flowed well, was easy to digest, was accessible, was interactive, and was an appropriate amount. Participants also appreciated being able to practice their communication skills. A few participants found that the additional links lengthened the education and desired more specific detail on the different types of COVID-19 vaccines.
      Figure 2
      Figure 2Feedback on the educational modules. The response is to the question: “How helpful were the vaccine modules to…,” with a scale of 1, not at all helpful; 2, slightly helpful; 3, somewhat helpful; 4, mostly helpful; and 5, very helpful.
      In reflecting on the educational curriculum in the follow-up survey, we received responses from 44 participants. Nearly all (98%) had had the opportunity to discuss COVID-19 vaccines with patients and their parents. For most participants, the modules decreased apprehension some or significantly (Table 5).
      Table 5Effect of Curriculum on Apprehension to Discuss COVID-19 Vaccine (n = 44)
      Did modules affect your level of apprehension

      to discuss COVID-19 vaccine?
      N (%)
      Increased apprehension significantly1 (2%)
      Increased apprehension some2 (5%)
      No change in my apprehension4 (9%)
      Decreased apprehension some25 (57%)
      Decreased apprehension significantly12 (27%)

      Discussion

      Pediatricians are well versed in discussing vaccines with families, as immunizations are a key tenet of pediatric preventive care. The global COVID-19 pandemic, vaccines using a novel messenger RNA platform, and the first large-scale deployment of vaccines through the emergency use authorization mechanism have all created barriers for pediatricians in having effective COVID-19 vaccine discussions with families. In addition, vaccine hesitancy is a global concern, and a survey of parents in early 2021 showed a majority are currently unlikely to or unsure about having their children vaccinated against COVID-19.
      • Szilagyi PG
      • Shah MD
      • Delgado JR
      • et al.
      Parents’ intentions and perceptions about COVID-19 vaccination for their children: results from a national survey.
      ,
      • Kempe A
      • Saville AW
      • Albertin C
      • et al.
      Parental hesitancy about routine childhood and influenza vaccinations: a national survey.
      ,
      • Opel DJ
      • Heritage J
      • Taylor JA
      • et al.
      The architecture of provider-parent vaccine discussions at health supervision visits.
      Given the anticipated challenges, five pediatric institutions successfully collaborated to deploy a timely, accessible online curriculum to improve pediatric providers’ abilities to address COVID-19 vaccine acceptance. Providers who completed the curriculum demonstrated significant knowledge improvement as well as increased confidence scores. In addition, the large effect sizes demonstrated the impact this curriculum delivered in knowledge and confidence.
      Since clinicians, especially pediatricians, remain a trusted source of information during the pandemic, they needed to be informed in a timely manner so that the adoption of the COVID-19 vaccine for a pediatric population could be accelerated. Historically, pediatricians have been familiar with vaccine counseling and conversations with vaccine-hesitant patients with routine vaccinations.
      • Kempe A
      • Saville AW
      • Albertin C
      • et al.
      Parental hesitancy about routine childhood and influenza vaccinations: a national survey.
      However, the educational content in this study leveraged existing research on how to promote vaccine uptake generally, such as onsite vaccination, and demonstrated how some of these behavioral approaches can be applied to COVID-19 vaccine communication and administration.
      • Opel DJ
      • Heritage J
      • Taylor JA
      • et al.
      The architecture of provider-parent vaccine discussions at health supervision visits.
      Providers are inundated with rapidly evolving COVID-19 vaccine information. This curriculum provided succinctly summarized, up-to-date, and interactive vaccine information in addition to leveraging simulation to practice motivational interviewing techniques
      • Oduwole EO
      • Pienaar ED
      • Mahomed H
      • Wiysonge CS.
      Current tools available for investigating vaccine hesitancy: a scoping review protocol.
      and was well received, with most participants reporting the modules to be helpful. While most of the results were positive, notably, in the follow-up survey participants became less confident about talking about risks of COVID-19 vaccines. This could be due to the fact that many of the participants completed the modules in summer 2021 with the emergence of myocarditis occurring after vaccine administration and lack of clarity around its significance.
      This work has several general limitations. First, our overall sample size was small despite multicenter and multipronged targeting efforts and the addition of enduring education credit. Each of the five institutions used similar language and recruiting methods; however, we did not track the overall number of possible participants solicited so can't report a formal response rate. Second, there were significant time and access barriers for the targeted population, as it was a 3-hour commitment and a new platform for most participants. In addition, due to the pandemic, many pediatric providers were overwhelmed catching up on missed routine visits, further limiting their time. These factors may have led to the significant drop-off resulting in a lower-than-expected completion rate. However, while participation dropped off, large effect sizes indicate that the sample size was sufficient to detect a meaningful difference between groups. Third, this work, including the educational content for best practices in discussing vaccines, was based on what was known at the time and built on other vaccine acceptance literature, though not specifically related to COVID-19 vaccines. We presumed there may be some crossover in concepts and some new challenges. Finally, this work reports preliminary findings of knowledge and satisfaction; nevertheless, it is hoped that widespread availability could have a broader impact that could be measured if we were able to perform longer-term follow-up. Although our participants reported a significant change from mostly confident to very confident in their abilities in some areas, it is unclear how this confidence will translate to the clinical setting; future plans include connecting with participants to determine this impact.
      The study design and structure also contributed limitations to this study. First, due to limited time and conscious efforts to minimize contact with participants in a research study, and the structure of the learning management system, we were not able to inform learners of updates to content. This will be a future area for improvement, as we hope it will allow for targeted updates based on timing of participation. Second, our design affected the ability to obtain longer-term follow-up data from the learners. Finally, our lack of a control group limits our ability to attribute the findings solely to the modules.

      Conclusion

      The work described in this paper reflects an agile deployment of an innovative curricular initiative focused on getting up-to-date information to front-line providers responsible for having complex conversations about COVID-19 vaccine decision-making and ensuring these vaccines are available in a thoughtful manner. The project demonstrated that an online platform could be developed, modified, and accessed by providers and that those who completed the experience had sustained, increased improvement in confidence. In the current era and for future pandemics, the platform that was developed and deployed could be adapted for continued impact.

      What's New

      Before authorization of COVID-19 vaccines for children ≥12 years, we formed a collaborative to develop and evaluate an educational curriculum for clinicians to promote vaccine acceptance. The curriculum increased clinician knowledge and led to sustained confidence in discussing COVID-19 vaccines.

      Conflict of Interest

      The authors have no conflicts of interest relevant to this article to disclose.

      Acknowledgments

      This project was supported by internal funds from each of the five collaborating institutions. We sincerely appreciate the technical expertise of Kurt Sidenstick and Melissa Madden in developing and executing the online modules. We thank Mark Adler, MD, Mark Baker, MD, Andrea Bradley-Ewing, MPA, MA, Ellie Hamburger, MD, Annika Hofstetter, MD, PhD, MPH, Chris Kennedy, MD, David Kimberlin, MD, Steve Krug, MD, Karen Mangold, MD, MEd, Shaina Newman, PA-C, and Kim Stone, MD, for their assistance in developing and testing the modules, communicating with participants, and executing the project.

      Appendix. Supplementary Data

      References

        • Posfay-Barbe KM
        • Wagner N
        • Gauthey M
        • et al.
        COVID-19 in children and the dynamics of infection in families.
        Pediatrics. 2020; 146e20201576https://doi.org/10.1542/peds.2020-1576
        • Wanga V
        • Gerdes ME
        • Shi DS
        • et al.
        Characteristics and clinical outcomes of children and adolescents aged <18 years hospitalized with COVID-19—six hospitals, United States, July–August 2021.
        MMWR Morb Mortal Wkly Rep. 2021; 70: 1766-1772https://doi.org/10.15585/mmwr.mm705152a3
        • Chu VT
        • Yousaf AR
        • Chang K
        • et al.
        Georgia Camp Investigation Team. Household transmission of SARS-CoV-2 from children and adolescents.
        N Engl J Med. 2021; 385: 954-956https://doi.org/10.1056/NEJMc2031915
        • Paul LA
        • Daneman N
        • Schwartz KL
        • et al.
        Association of age and pediatric household transmission of SARS-CoV-2 infection.
        JAMA Pediatr. 2021; 175: 1151-1158https://doi.org/10.1001/jamapediatrics.2021.2770
      1. American Academy of Pediatrics. Children and COVID-19: State-level data report. Accessed November 2, 2021. https://www.aap.org/en/pages/2019-novel-coronavirus-covid-19-infections/children-and-covid-19-state-level-data-report. Updated 2021.

      2. US Food and Drug Administration. Coronavirus (COVID-19) update: FDA authorizes Pfizer-BioNTech COVID-19 vaccine for emergency use in adolescents in another important action in fight against pandemic. Accessed May 10, 2021. https://www.fda.gov/news-events/press-announcements/coronavirus-covid-19-update-fda-authorizes-pfizer-biontech-covid-19-vaccine-emergency-use.

      3. US Food and Drug Administration. FDA authorizes Pfizer-BioNTech COVID-19 vaccine for emergency use in children 5 through 11 years of age. Accessed October 29, 2021. https://www.fda.gov/news-events/press-announcements/fda-authorizes-pfizer-biontech-covid-19-vaccine-emergency-use-children-5-through-11-years-age.

        • Centers for Disease Control and Prevention
        Understanding mRNA COVID-19 vaccines.
        Accessed. October 26, 2021;
        • Centers for Disease Control and Prevention
        COVID-19 vaccination.
        Accessed. October 26, 2021;
      4. Hamel L, Lopes L, Kearney A, et al. KFF COVID-19 vaccine monitor: Winter 2021 update on parents’ views of vaccines for kids. Kaiser Family Foundation. December 9, 2021. https://www.kff.org/coronavirus-covid-19/poll-finding/kff-covid-19-vaccine-monitor-winter-2021-update-on-parents-views-of-vaccines/

        • Marotta S
        • McNally VV.
        Increasing vaccine confidence through parent education and empowerment using clear and comprehensible communication.
        Acad Pediatr. 2021; 21: S30-S31https://doi.org/10.1016/j.acap.2021.01.016
        • Freed GL
        • Clark SJ
        • Butchart AT
        • Singer DC
        • Davis MM.
        Sources and perceived credibility of vaccine-safety information for parents.
        Pediatrics (Evanston). 2011; 127: S107-S112https://doi.org/10.1542/peds.2010-1722P
        • Szilagyi PG
        • Shah MD
        • Delgado JR
        • et al.
        Parents’ intentions and perceptions about COVID-19 vaccination for their children: results from a national survey.
        Pediatrics (Evanston). 2021; 148https://doi.org/10.1542/peds.2021-052335
      5. Centers for Disease Control and Prevention. COVID-19 Vaccine Training Modules. https://www2.cdc.gov/vaccines/ed/covid19/. Accessed April 4, 2022.

        • Wilson PM
        • Petticrew M
        • Calnan MW
        • Nazareth I.
        Disseminating research findings: what should researchers do? A systematic scoping review of conceptual frameworks.
        Implement Sci. 2010; 5: 91https://doi.org/10.1186/1748-5908-5-91
        • Winkler JD
        • Lohr KN
        • Brook RH.
        Persuasive communication and medical technology assessment.
        Arch Intern Med. 1985; 145: 314-317
        • Lasswell HD.
        The structure and function of communication in society.
        in: Bryson L The Communication of Ideas. Harper and Row, New York1948: 37-51
      6. McGuire W. McGuire's classic input-output framework for constructing persuasive messages. In: Rice RE, Atkin CK, eds. Public Communication Campaigns (4th ed.). Washington, DC: Sage; 2013: 133–145.

        • Thomas PA
        • Kern DE
        • Hughes MT
        • Chen BY
        Curriculum Development for Medical Education: A Six-Step Approach.
        Johns Hopkins University Press, Baltimore2016
        • Opel DJ
        • Lo B
        • Peek ME.
        Addressing mistrust about COVID-19 vaccines among patients of color.
        Ann Intern Med. 2021; 174: 698-700https://doi.org/10.7326/M21-0055
        • Koski K
        • Lehto JT
        • Hakkarainen K.
        Simulated encounters with vaccine-hesitant parents: arts-based video scenario and a writing exercise.
        J Med Educ Curric Dev. 2018; 52382120518790257https://doi.org/10.1177/2382120518790257
        • Nold L
        • Deem MJ.
        A simulation experience for preparing nurses to address refusal of childhood vaccines.
        J Nurs Educ. 2020; 59: 222-226https://doi.org/10.3928/01484834-20200323-09
        • Oduwole EO
        • Pienaar ED
        • Mahomed H
        • Wiysonge CS.
        Current tools available for investigating vaccine hesitancy: a scoping review protocol.
        BMJ Open. 2019; 9e033245https://doi.org/10.1136/bmjopen-2019-033245
        • Pahud B
        • Williams SE
        • Lee BR
        • et al.
        A randomized controlled trial of an online immunization curriculum.
        Vaccine. 2020; 38: 7299-7307https://doi.org/10.1016/j.vaccine.2020.09.043
        • Real FJ
        • DeBlasio D
        • Beck AF
        • et al.
        A virtual reality curriculum for pediatric residents decreases rates of influenza vaccine refusal.
        Acad Pediatr. 2017; 17: 431-435https://doi.org/10.1016/j.acap.2017.01.010
        • Shalansky RA
        • Wu M
        • Shen SC
        • et al.
        Evaluation of a pilot immunization curriculum to meet competency training needs of medical residents.
        BMC Med Educ. 2020; 20: 442https://doi.org/10.1186/s12909-020-02349-1
        • World Health Organization
        Timeline: WHO's COVID-19 response.
        Accessed. October 7, 2021;
      7. US Food and Drug Administration. Commissioner of the FDA takes key action in fight against COVID-19 by issuing emergency use authorization for first COVID-19 vaccine. December 11, 2020. https://www.fda.gov/news-events/press-announcements/fda-takes-key-action-fight-against-covid-19-issuing-emergency-use-authorization-first-covid-19.

      8. National Foundation for Infectious Diseases. Coronaviruses. July 27, 2021. https://www.nfid.org/infectious-diseases/coronaviruses/.

        • Centers for Disease Control and Prevention
        SARS-COV-2 variant classifications and definitions.
        Accessed. October 13, 2021;
      9. Woodward A. One chart shows how well COVID-19 vaccines work against the 3 most worrisome coronavirus variants. Business Insider. March 12, 2021. https://www.businessinsider.com/covid-19-vaccines-efficacy-variants-south-africa-uk-brazil-2021-3

      10. Center for Biologics Evaluation and Research. Emergency use authorization for vaccines explained. Accessed October 13, 2021. https://www.fda.gov/vaccines-blood-biologics/vaccines/emergency-use-authorization-vaccines-explained.

      11. US Department of Health and Human Services. Register today! Become a part of the expanded COVID-19 vaccine workforce. Accessed October 13, 2021. https://www.phe.gov/emergency/events/COVID19/COVIDvaccinators/Pages/register.aspx.

      12. Centers for Disease Control and Prevention. Administration overview for Pfizer-Biontech COVID-19 vaccine. August 25, 2021. https://www.cdc.gov/vaccines/covid-19/info-by-product/pfizer/index.html.

        • Centers for Disease Control and Prevention
        Appendix B: Triage of people with a history of allergies or allergic reactions.
        Accessed. October 14, 2021;
      13. CMS. Covid-19 vaccines and monoclonal antibodies. Accessed October 14, 2021. https://www.cms.gov/medicare/medicare-part-b-drug-average-sales-price/covid-19-vaccines-and-monoclonal-antibodies.

        • Bryant KA
        • Wesley GC
        • Wood JA
        • Hines C
        • Marshall GS.
        Use of standardized patients to examine physicians’ communication strategies when addressing vaccine refusal: a pilot study.
        Vaccine. 2009; 27: 3616-3619https://doi.org/10.1016/j.vaccine.2009.03.048
        • Loehr J
        • Savoy M.
        Strategies for addressing and overcoming vaccine hesitancy.
        Am Fam Physician. 2016; 94: 94-96
      14. Olson O, Berry C, Kumar N. Addressing parental vaccine hesitancy towards childhood vaccines in the United States: a systematic literature review of communication interventions and strategies. Vaccines (Basel). 2020;8(4):590. doi:10.3390/vaccines8040590.

        • Henrikson NB
        • Opel DJ
        • Grothaus L
        • et al.
        Physician communication training and parental vaccine hesitancy: a randomized trial.
        Pediatrics. 2015; 136: 70-79https://doi.org/10.1542/peds.2014-3199
        • Cook DA
        • Beckman TJ.
        Current concepts in validity and reliability for psychometric instruments: theory and application.
        Am J Med. 2006; 166: e7-e166
        • Kerby DS.
        The simple difference formula: an approach to teaching nonparametric correlation.
        Comprehensive Psychol. 2014; 3https://doi.org/10.2466/11.IT.3.1
        • Ruscio J.
        A probability-based measure of effect size: robustness to base rates and other factors.
        Psychol Methods. 2008; 13: 19-30https://doi.org/10.1037/1082-989X.13.1.19
        • Kempe A
        • Saville AW
        • Albertin C
        • et al.
        Parental hesitancy about routine childhood and influenza vaccinations: a national survey.
        Pediatrics (Evanston). 2020; 146e20193852https://doi.org/10.1542/peds.2019-3852
        • Opel DJ
        • Heritage J
        • Taylor JA
        • et al.
        The architecture of provider-parent vaccine discussions at health supervision visits.
        Pediatrics (Evanston). 2013; 132: 1037-1046https://doi.org/10.1542/peds.2013-2037