Advertisement

How Prepared Are They? Pediatric Boot Camps and Intern Performance

Open AccessPublished:May 13, 2022DOI:https://doi.org/10.1016/j.acap.2022.05.005

      Abstract

      Objective

      To determine whether participation in a pediatric boot camp during medical school was associated with higher intern performance. Secondary objectives were to determine whether participation in general boot camps, pediatric subinternships or pediatric electives was associated with higher performance.

      Methods

      Intern surveys and faculty performance assessments during early internship were collected from a convenience sample of pediatric residency programs. Interns completed a survey regarding participation in medical school boot camps, pediatric subinternships and pediatric electives. Faculty assessed intern performance on selected Milestone-based subcompetencies on a 5-point scale following each intern's initial inpatient rotation and results were compared between groups.

      Results

      Seventeen pediatric residency programs participated. Two hundred eighty-seven interns completed the survey (69%), and faculty completed assessments on 71% of these interns. Of interns with complete faculty assessments (n = 198), 25% participated in 5 or more days of pediatric boot camp, 30% in general boot camp, and 45% in no boot camp. There were no educationally significant associations between participation in 5 or more days of pediatric boot camp, general boot camp, subinternships, or electives and intern performance. Interns completing at least 10 days of pediatric boot camp (n = 25) had slightly higher ratings for incorporating feedback and engaging in help-seeking behavior during June and July only.

      Conclusions

      Participation in pediatric boot camps, general boot camps, pediatric subinternships or electives was not associated with substantially higher intern performance as measured by selected Milestone subcompetencies. Pediatric educators should carefully consider boot camp curricula and anticipated outcomes associated with boot camp participation.

      Keywords

      What's New
      This is the first study in pediatrics to examine faculty assessment of intern performance after participation in medical student boot camps.
      The transition from medical student to intern is challenging. Program directors have identified several domains in which interns struggle including lack of self-reflection and improvement, poor organizational skills, underdeveloped professionalism and weak medical knowledge.
      • Lyss-Lerman P
      • Teherani A
      • Aagaard E
      • et al.
      What training is needed in the fourth year of medical school? Views of residency program directors.
      Despite these residency program director concerns, most medical schools provide a very flexible fourth year curriculum.
      • Walling A
      • Merando A
      The fourth year of medical education. A literature review.
      • Goldfarb S
      • Morrison G.
      The 3-year medical school–change or shortchange?.
      • Kanter SL.
      How to win an argument about the senior year of medical school.
      Although many students consider preparation for residency the most important function of their final year of medical school, many also feel that having a broad educational experience in a variety of medical fields is similarly important.
      • Benson N
      • Stickle T
      • Raszka W.
      Going “Fourth” from medical school: fourth-year medical students’ perspectives on the fourth year of medical school.
      To address both the variability of the fourth year of medical school experience and expectations residency programs have of interns, some professional organizations have recommended residency preparatory courses (boot camps)
      • Teo AR
      • Harleman E
      • O'Sullivan P
      • et al.
      The key role of a transition course in preparing medical students for internship.
      • Reddy ST
      • Chao J
      • Carter JL
      • et al.
      Alliance for clinical education perspective paper: recommendations for redesigning the “final year” of medical school.
      American Board of Surgery
      Statement on surgical pre-residency preparatory courses.
      before graduation from medical school to further prepare students for the transition from medical school to internship.
      Although boot camps are frequently offered at medical schools, there are few objective studies of intern performance following participation. Some studies of boot camps have shown an improvement in skills, knowledge, or confidence among graduating students.
      • Neylan CJ
      • Nelson EF
      • Dumon KR
      • et al.
      Medical school surgical boot camps: a systematic review.
      • Blackmore C
      • Austin J
      • Lopushinsky S
      • et al.
      Effects of postgraduate medical education "boot camps" on clinical skills, knowledge, and confidence: a meta-analysis.
      • Burns R
      • Adler M
      • Mangold K
      • et al.
      Brief boot camp for 4th-year medical students entering into pediatric and family medicine residencies.
      • Green C
      • Vaughn C
      • Wyles S
      • et al.
      Evaluation of a surgery-based adjunct course for senior medical students entering surgical residencies.
      • Minter R
      • Amos K
      • Bentz M
      • et al.
      Transition to surgical residency: a multi-institutional study of perceived intern preparedness and the effect of a formal residency preparatory course in the fourth year of medical school.
      • Okusanya O
      • Kornfield Z
      • Reinke C
      • et al.
      The effect and durability of a pregraduation boot cAMP on the confidence of senior medical student entering surgical residencies.
      • Bontempo L
      • Frayha N
      • Dittmar P.
      The internship preparation camp at the University of Maryland.
      • Pete Devon E
      • Tenney-Soeiro R
      • Ronan J
      • et al.
      A pediatric preintern boot camp: program development and evaluation informed by a conceptual framework.
      • Antonoff M
      • Swanson J
      • Green C
      • et al.
      The significant impact of a competency-based preparatory course for senior medical students entering surgical residency.
      • Costich M
      • Finkel M
      • Friedman S
      • et al.
      Transition-to-residency: pilot innovative, online case-based curriculum for medical students preparing for pediatric internships.
      However, most rely on self-reported confidence or preparedness and do not include assessments of performance during internship. In one study, participants in a 5-day elective pediatric boot camp demonstrated a higher performance in skills and reported a higher level of preparedness in targeted domains immediately following the course.
      • Burns R
      • Adler M
      • Mangold K
      • et al.
      Brief boot camp for 4th-year medical students entering into pediatric and family medicine residencies.
      In another, students reported a higher level of perceived self-efficacy for nearly all skills at the end of a 5-day boot camp and this persisted for 6 months.
      • Pete Devon E
      • Tenney-Soeiro R
      • Ronan J
      • et al.
      A pediatric preintern boot camp: program development and evaluation informed by a conceptual framework.
      These studies suggest a positive association of medical school boot camps with intern performance, yet there is a lack of published data using faculty-based performance assessments during early pediatric internship related to boot camp participation.
      This study investigated whether self-reported participation in a pediatric boot camp during the final year of medical school is associated with higher performance during initial intern inpatient rotations (general pediatrics/subspecialty or neonatal intensive care unit) as reflected on Accreditation Council for Graduate Medical Education (ACGME) Milestones-based subcompetency ratings

      General Pediatrics Reported Milestones. Accreditation Council for Graduate Medical Education. 2015. Available at: https://www.acgme.org/globalassets/pdfs/milestones/crosswalkpediatricsreportingmilestones.pdf. Accessed February 22, 2022.

      ,
      • Englander R
      • Cameron T
      • Ballard A
      • et al.
      Toward a common taxonomy of competency domains for the health professions and competencies for physicians.
      compared with peers who did not participate in a pediatric boot camp. A secondary aim of the study was to determine the association between participation in general boot camps, pediatric subinternships and pediatric electives and performance in early internship.

      Methods

      A convenience sample of Program Directors (PDs) was recruited for this cross-sectional study in March 2019 at the annual Association of Pediatric Program Directors (APPD) meeting through an announcement at the conference and through an email invitation sent to PDs from the APPD Longitudinal Educational Assessment Research Network.
      • Schwartz A
      • King B
      • Mink R
      • et al.
      The emergence and spread of practice-based medical education research networks.
      Programs were also recruited through the Council on Medical Student Education in Pediatrics Boot Camp Subgroup; these educators recruited PDs from their respective institutions to participate. Educators in multiple roles within undergraduate to graduate medical education at different institutions joined to form the APPD Longitudinal Educational Assessment Research Network Boot Camp Study Group. Members included the study authors and other educators nationally recognized as leaders in boot camp curriculum design and implementation.
      • Hartke A
      • Devon EP
      • Burns R
      • et al.
      Building a boot camp: pediatric residency preparatory course design workshop and took kit.
      Authors designed the study, and members of the study group facilitated implementation of the study at individual institutions.
      In July 2019, all interns from participating programs were invited by their respective PDs to complete a survey via email regarding boot camp course participation and boot camp course characteristics including length, timing, and duration of pediatric versus general components (Appendix A). Interns were also asked to report on the number of pediatric subinternships and pediatric electives (0–5) during their postclerkship clinical rotations in medical school.
      For this study, “boot camp” was defined as a course during the postclerkship phase of medical school designed to enhance preparation for residency. Participation in at least 5 days of pediatric-specific coursework was considered a pediatric boot camp, and participation of 0 to 4 days of pediatric-specific coursework, including within a general boot camp, was considered a general boot camp. The decision to make 5 days or more the accepted standard for “pediatric boot camp” participation was based on published national data on the distribution of boot camp duration
      • Hartke A
      • Devon EP
      • Burns R
      • et al.
      Building a boot camp: pediatric residency preparatory course design workshop and took kit.
      and by consensus of the authors.
      During the initial months of the intern academic year, PDs from participating programs enlisted faculty who worked with each intern to complete a unique assessment of specific Milestone-based subcompetencies (Table 1) following their initial inpatient rotation (Appendix B). One faculty member completed the assessment form for each intern.
      Table 1Milestone Subcompetencies Included on Faculty Assessment Surveys
      PC2Organize/prioritize responsibilities to provide patient care that is safe, effective, and efficient
      PC3Provide transfer of care that ensures seamless transitions
      PC4Make informed diagnostic and therapeutic decisions that result in optimal clinical judgment
      SBP3work in inter-professional teams to enhance patient safety and improve patient care quality
      PBLI4Incorporate formative evaluation feedback into daily practice
      PROF4Self-awareness of knowledge, skills, and emotional limitations to engage in help-seeking behavior
      ICS1Communicate effectively with patients, families, and the public, as appropriate, across a broad range of socioeconomic and cultural backgrounds
      PC indicates patient care; SBP, systems-based practice; PBLI, practice-based learning and improvement; PROF, professionalism; and ICS, interpersonal skills and communication.
      Milestones subcompetencies selected for use in intern assessments, from ACGME General Pediatrics Reported Milestones.
      Faculty surveys aligned with the current framework of ACGME Milestone-based assessments already being used for residency programs using identical language and a similar format. The Milestone-based subcompetencies selected for faculty assessments were those most reflective of content in published studies of pediatric boot camps and in authors’ experiences as boot camp course directors.
      • Lyss-Lerman P
      • Teherani A
      • Aagaard E
      • et al.
      What training is needed in the fourth year of medical school? Views of residency program directors.
      ,
      • Teo AR
      • Harleman E
      • O'Sullivan P
      • et al.
      The key role of a transition course in preparing medical students for internship.
      ,
      • Burns R
      • Adler M
      • Mangold K
      • et al.
      Brief boot camp for 4th-year medical students entering into pediatric and family medicine residencies.
      ,
      • Pete Devon E
      • Tenney-Soeiro R
      • Ronan J
      • et al.
      A pediatric preintern boot camp: program development and evaluation informed by a conceptual framework.
      ,
      • Hartke A
      • Devon EP
      • Burns R
      • et al.
      Building a boot camp: pediatric residency preparatory course design workshop and took kit.
      The 7 subcompetencies reflect patient care skills (prioritization, handoff, clinical decision-making) and interpersonal skills (incorporating formative feedback, help-seeking behavior, interprofessional teamwork, communication). The interns were rated on a 5-point scale with 4 half-points. Incentives for completing the surveys were provided as gift cards to both interns ($15) and PDs ($50). All participating institutions obtained institutional review board approval from their respective institutions. Surveys and faculty assessments were de-identified and only an identifying number assigned by the residency program was available to investigators to correlate responses.
      Primary analysis included a comparison of intern performance based on faculty assessment during the first inpatient rotation of internship (general pediatrics/subspecialty or neonatal intensive care unit) for those who completed a pediatric boot camp with those completing no boot camp. Secondary analyses were performed for interns who completed a general boot camp with those completing no boot camp as well as for performance related to the number of pediatric subinternships and to the number of pediatric postclerkship electives completed during the fourth year.
      We fit linear mixed models to each individual milestone subcompetency rating. Models included a fixed effect of pediatric boot camp participation, a random effect of program to account for clustering in programs, Step 2 CK score (intern-level, centered at 240), number of postclerkship clinical pediatric electives and pediatric subinternships, and whether the residency program was at the same or different institution as the learner's medical school (vs unable to determine) as covariates. Performance data were analyzed in aggregate for interns whose first inpatient month was in June to October as well as separately for interns whose first inpatient month was in June/July.
      We deemed a difference of 0.5 points on the 5-point Milestone rating scales to be the minimal educationally significant difference since the milestone scale allows reporting up to 0.5 points and a change of 0.5 would move the learner to a different competency rating. Assuming a within-program standard deviation of 0.80, a between-program standard deviation of 1.25,
      • Hu K
      • Hicks PJ
      • Margolis M
      • et al.
      Reported Pediatrics Milestones (mostly) measure program, not learner performance.
      and the proportion of interns with at least 5 days of pediatric-specific boot camp coursework between 25% and 65%, a sample size of 280 interns from 18 to 22 programs would provide at least 80% power to detect a 0.5-point difference on a 5-point rating scale between interns with and without pediatric boot camp participation.

      Results

      All residency programs indicating an initial interest participated, giving a study population of 17 pediatric residency programs. All 413 interns from these programs were invited by their PD's to participate. Program size ranged from 7 to 57 interns per class and were geographically diverse (Table 2). Two hundred and eighty-seven interns returned surveys for a response rate of 69% (median 76%; interquartile range 53%–94%; range 19%–100%). Faculty completed assessments on 257 interns for a response rate of 62% (median 64%; interquartile range 44%–93%; range 19%–100%). Out of these responses, there were 205 completed intern surveys with paired faculty assessments; Seventy-one percent of participating interns had paired data giving an adjusted response rate for all interns of 50%. All analytic variables were present in 198 out of these 205 surveys (Figure). The study was halted after 1 year as the lockdown in March 2020 associated with the coronavirus pandemic led most medical schools to quickly convert to virtual learning formats. Since boot camps typically take place in March or April, the authors felt that this change would lead to boot camp courses quite different than in previous years and this could skew subsequent intern performance data.
      Table 2Participating Institutions by APPD Geographic Region and Size
      APPD Regions:Small ProgramsMedium-Sized ProgramsLarge Programs
      Mid-AmericaAkron Children's Hospital

      MSU/Helen DeVos Children's Hospital
      Rainbow Babies and Children's Hospital

      Nationwide Children's Hospital
      Mid-AtlanticChildren's Hospital of Philadelphia

      Children's National Medical Center
      MidwestRush University Children's HospitalLurie Children's Hospital of Chicago, Feinberg School of Medicine
      NortheastUniversity of Vermont Children's Hospital
      SoutheastInova Children's HospitalUniversity of Tennessee Health Science Center

      Virginia Commonwealth University
      SouthwestUniversity of Texas at Houston
      WesternUniversity of Hawaii

      University of California Davis Children's Hospital

      University of Nevada Las Vegas, Children's Hospital of Nevada
      Seattle Children's Hospital

      APPD indicates Association of Pediatric Program Directors.
      Participating programs by APPD Geographic Region and Size.
      (Residents per class: small = 15 or fewer; medium = 16–30; large = more than 30).
      From paired intern surveys with all variables, 25% of interns reported participating in a pediatric boot camp (n = 49), 30% in a general boot camp (n = 59), and 45% in no boot camp (n = 90). For those who participated in a pediatric boot camp, the duration was 5 to 9 days in 49% (n = 24) and 10 or more days in 51% (n = 25). The mean length of the pediatric component of boot camps was 9.9 days for a pediatric boot camp (standard deviation 4.8; range 5–20 days) and 1.5 days for a general boot camp (standard deviation 1.2; range 0–4 days). Boot camp participation, type of boot camp, and length of boot camp did not differ significantly between interns with and without faculty surveys. Interns had participated in a mean of 1.7 pediatric subinternships (range 0–5) and 2.6 postclerkship pediatric electives (range 0–5).
      Examination of Milestone subcompetency performance ratings for interns completing pediatric boot camps (5 or more days) revealed higher ratings that were statistically but not educationally significant for one subcompetency, “Self-awareness of knowledge, skills, and emotional limitations to engage in help-seeking behavior” with ratings that were 0.28 points (P = .04) and 0.39 points (P = .049) higher for interns with their first inpatient month during June to October (Table 3) and for those starting in June or July (Table 4), respectively. Participation in general boot camps (<5 days pediatric specific content) did not reveal any statistically significant associations with intern performance (Tables 3 and 4). Post hoc analysis of intern performance for those interns who had completed 10 days or more of pediatric boot camp (n = 25) revealed a higher performance for 2 interpersonal skills-related subcompetencies, “Incorporate formative evaluation feedback into daily practice” and “Self-awareness of knowledge, skills, emotional limitations to engage in help-seeking behavior” that were both statistically and educationally significant (B = 0.50, P < .01; and B = 0.57, P < .01 on a 5-point scale, respectively) for interns with their first inpatient rotation in June or July (Table 4).
      Table 3Association Between Participation in Postclerkship Pediatric Courses and Performance on First Inpatient Rotation as an Intern (n = 198)
      Milestone Subcompetencies
      Patient CareInterpersonal
      PC2

      Organization Prioritization
      PC3

      Handoffs
      PC4

      Clinical Decision-Making
      PBLI4

      Incorporating Feedback
      PROF4

      Help-Seeking Behavior
      SBP3

      Interprofessional Teams
      ICS1

      Communication
      <5 Days pediatric boot campB*−0.020.070.010.100.110.070.05
      SE+0.120.100.010.110.130.110.12
      P.86.49.93.38.39.53.71
      ≥5 Days pediatric boot campB0.090.0040.740.150.28−0.07−0.03
      SE0.130.120.110.160.140.120.14
      P.50.97.51.21.04.96.81
      ≥10 Days pediatric boot campB0.130.0140.070.220.390.070.07
      SE0.150.140.130.150.170.140.16
      P.37.92.58.14.02.64.69
      Pediatric electiveB0.040.070.010.05−0.0040.040.04
      SE0.030.030.030.030.040.030.04
      P.26.03.67.11.92.21.31
      Pediatric subinternshipB0.100.080.100.040.050.060.08
      SE0.050.050.040.050.060.050.05
      P.06.10.02.42.41.18.12
      PC indicates patient care; PBLI, practice-based learning and improvement; PROF, professionalism; SBP, systems-based practice; and ICS, interpersonal skills and communication.
      Specific performance data were compared for interns who completed a given postclerkship student rotation (pediatric elective, pediatric subinternship, or boot camp with pediatric specific content of <5, ≥5, or ≥10 days) with interns who had not completed that rotation as a student. Performance data were collected after the first inpatient intern rotation (general/subspecialty pediatrics or NICU) and is shown for all months in aggregate.
      Bold indicates statistically significant.
      B* = unstandardized regression coefficient; SE+ = standard error of B.
      Table 4Association Between Participation in Postclerkship Pediatric Courses and Intern Performance on Early (June–July) First Inpatient Rotation (n = 114)
      Milestone Subcompetencies
      Patient CareInterpersonal
      PC2

      Organization Prioritization
      PC3

      Handoffs
      PC4

      Clinical Decision-Making
      PBLI4

      Incorporating Feedback
      PROF4

      Help-Seeking Behavior
      SBP3

      Interprofessional Teams
      ICS1

      Communication
      <5 Days pediatric boot campB*0.060.180.080.160.100.080.03
      SE+0.140.130.120.150.180.150.16
      P.69.17.53.27.55.61.87
      ≥5 Days pediatric boot campB0.150.130.10.320.390.210.19
      SE0.150.150.140.170.20.170.18
      P.32.39.47.05.049.22.3
      ≥10 Days pediatric boot campB0.260.230.130.50.570.330.34
      SE0.180.170.160.190.230.20.21
      P.15.19.44.01.01.1.12
      Pediatric electiveB0.050.10.030.060.010.040.04
      SE0.040.040.040.050.060.050.05
      P.21.02.53.21.79.41.47
      Pediatric subinternshipB0.170.130.170.060.050.120.12
      SE0.060.060.060.070.080.070.07
      P.006.03.002.34.52.06.12
      PC indicates patient care; PBLI, practice-based learning and improvement; PROF, professionalism; SBP, systems-based practice; and ICS, interpersonal skills and communication.
      Specific performance data were compared for interns who completed a given postclerkship student rotation (pediatric elective, pediatric subinternship, or boot camp with pediatric specific content of <5, ≥5, or ≥10 days) with interns who had not completed that rotation as a student. Performance data were collected after the first inpatient intern rotation (general/subspecialty pediatrics or NICU) and is shown for interns who had this rotation in June or July.
      Bold indicates statistically significant; underlined = educationally significant.
      B* = unstandardized regression coefficient; SE+ = standard error of B.
      In comparing interns who completed boot camps that differed in duration of pediatric content, there were no statistically or educationally significant associations for interns who had completed pediatric boot camp compared with general boot camp on performance ratings. Similarly there was no significant association between interns who participated in any boot camp (pediatric or general) with no boot camp participation on performance ratings.
      Participation in postclerkship clinical pediatric electives showed statistically but not educationally significant differences in ratings in the patient care milestone for handoffs (Tables 3 and 4). Participation in subinternships also revealed statistically but not educationally significant differences for patient care subcompetencies, with an increase of 0.10 to 0.17 points on a 5-point scale per subinternship completed with larger effects noted for interns completing their first rotation within the June to July (Table 4) compared to June to October time frame (Table 3). Post hoc analysis revealed that performance was not significantly higher in any subcompetencies unless at least 4 subinternships were completed (n = 8).
      There was no significant difference in Step 2 CK scores or total number of pediatric clinical rotations among students in the pediatric, general, or no boot camp groups. There was also no difference in faculty rating of any subcompetency between interns with paired data and those without, adjusting for program.

      Discussion

      In this study, interns who completed a pediatric boot camp of 5 days or more did not perform significantly higher on selected Milestone subcompetencies than interns who had not completed a pediatric boot camp. Interns who completed a pediatric boot camp of 10 days or more and had their first inpatient rotation in June or July had a slightly higher performance rating on subcompetencies related to incorporating formative feedback and self-awareness to engage in help-seeking behavior that was both statistically and educationally significant. There was no educationally significant association between intern performance and participation in general boot camps, pediatric subinternships, or pediatric electives in the fourth year.
      While some studies have demonstrated an improvement in students’ knowledge, skills, and self-reported confidence or preparedness directly after participation in medical school boot camps,
      • Neylan CJ
      • Nelson EF
      • Dumon KR
      • et al.
      Medical school surgical boot camps: a systematic review.
      ,
      • Burns R
      • Adler M
      • Mangold K
      • et al.
      Brief boot camp for 4th-year medical students entering into pediatric and family medicine residencies.
      • Green C
      • Vaughn C
      • Wyles S
      • et al.
      Evaluation of a surgery-based adjunct course for senior medical students entering surgical residencies.
      • Minter R
      • Amos K
      • Bentz M
      • et al.
      Transition to surgical residency: a multi-institutional study of perceived intern preparedness and the effect of a formal residency preparatory course in the fourth year of medical school.
      • Okusanya O
      • Kornfield Z
      • Reinke C
      • et al.
      The effect and durability of a pregraduation boot cAMP on the confidence of senior medical student entering surgical residencies.
      • Bontempo L
      • Frayha N
      • Dittmar P.
      The internship preparation camp at the University of Maryland.
      • Pete Devon E
      • Tenney-Soeiro R
      • Ronan J
      • et al.
      A pediatric preintern boot camp: program development and evaluation informed by a conceptual framework.
      • Antonoff M
      • Swanson J
      • Green C
      • et al.
      The significant impact of a competency-based preparatory course for senior medical students entering surgical residency.
      • Costich M
      • Finkel M
      • Friedman S
      • et al.
      Transition-to-residency: pilot innovative, online case-based curriculum for medical students preparing for pediatric internships.
      our study is the first to assess clinical performance during pediatric internship. Potential explanations for the lack of association for most of the subcompetencies is that boot camps expose participants to large amounts of medical content and skill practice in a very short time, the learning sessions are often not workplace-specific, and there is a gap in time, typically 3 to 4 months, before students apply these skills during internship. The slightly higher performance during June/July for two interpersonal skills-related subcompetencies following at least 10 days of pediatric boot camp may be consistent with the increase in student self-perceived confidence and preparedness demonstrated in prior studies.
      This study had several limitations. While overall there was a high response rate for both intern surveys and faculty assessments, there was a mismatch resulting in faculty assessments for only 71% of participating interns leaving many unpaired faculty/intern surveys. Although there are published models of recommended pediatric boot camp content,
      • Burns R
      • Adler M
      • Mangold K
      • et al.
      Brief boot camp for 4th-year medical students entering into pediatric and family medicine residencies.
      ,
      • Pete Devon E
      • Tenney-Soeiro R
      • Ronan J
      • et al.
      A pediatric preintern boot camp: program development and evaluation informed by a conceptual framework.
      ,
      • Hartke A
      • Devon EP
      • Burns R
      • et al.
      Building a boot camp: pediatric residency preparatory course design workshop and took kit.
      many of them are based on the Association of American Medical Colleges Entrustable Professional Activities,
      and no nationally accepted standard format for pediatric boot camps exists. Since we did not collect information about the medical school where interns participated in boot camps, we were unable to control for boot camp content or format. Although we expect that interns should able to recall the duration of their boot camp experience, we relied on self-report about participation in and characteristics of boot camps. The study team selected the Milestone-based subcompetencies for faculty assessments based on previously published data on pediatric boot camp content and their best estimate as national leaders in pediatric boot camp design and implementation; however, there may have been many additional relevant subcompetencies that were not assessed. We selected a Milestones-based assessment since that model is familiar to residency programs: however, it is possible that using the Association of American Medical Colleges Entrustable Professional Activities or other assessment tools for faculty assessment of interns rather than Milestones may have altered our findings.
      In addition, the study did not control for the amount of clinical time the faculty member spent with the intern, whether it was a composite assessment, or the time elapsed before faculty survey completion, all which could have affected quality or accuracy. Since faculty and intern surveys were deidentified, the study team was unable to control for individual faculty ratings although results were controlled by program. Similarly, the study did not ask whether the interns were higher- or lower-performing within their program; there may have been selection bias related to the students who chose to participate in boot camps, and boot camps may affect higher or lower-performing students differently. There may also have been selection bias related to those who responded to the intern surveys and also to the faculty assessments. Last, due to the coronavirus pandemic lockdowns in early spring 2020 and the pivot to virtual learning formats for boot camps, the study was halted after obtaining only 1 year of data. As such, we did not reach our anticipated 280 matched pairs and there is a possibility that additional data would have uncovered new associations with intern performance or hidden the small associations we found.
      If the ultimate goal of boot camps is to improve intern performance, our results suggest that current models may not be achieving these goals. This does not imply that boot camps do not have educational value, however. Key areas of focus to improve the undergraduate medical education (UME) to graduate medical education (GME) transition as summarized in the Coalition for Physician Accountability UME to GME Review Committee report
      • Lovell E
      • Mejicano G
      co-chairs
      The Coalition for Physician Accountability’s Undergraduate Medical Education-Graduate Medical Education Review Committee (UGRC): Recommendations for Comprehensive Improvement of the UME-GME Transition.
      include coaching, assessment following residency application submission, and UME to GME communication. These are all elements that could potentially be addressed within medical student specialty-specific transition to residency courses and warrant further study. Alignment of pediatric boot camp content with intern needs and gaps as determined by pediatric residency program directors followed by careful performance measurement will be instrumental in future study of residency preparation courses. Pediatric boot camps provide a unique opportunity to assess student skills and areas for improvement at the end of medical school, information that could be invaluable for residency program directors looking to provide individualized support and schedule planning for incoming interns.
      Given our modest positive study findings related to incorporating formative feedback and self-awareness to engage in help-seeking behavior following 10 days or more of pediatric boot camp for interns with their first inpatient rotation in June/July, it is possible that pediatric boot camps could help to address program director concerns related to interns’ lack of self-reflection and improvement.
      • Lyss-Lerman P
      • Teherani A
      • Aagaard E
      • et al.
      What training is needed in the fourth year of medical school? Views of residency program directors.
      Although our study focused exclusively on faculty assessment of ACGME Milestones-based subcompetencies, specialty-specific boot camps may be important in cultivating aspects of a developing physician that are more difficult to measure such as professional identity. Boot camps provide the opportunity to not only connect with medical school peers, but also to reflect on past experiences and share future excitement and anxiety. They also provide the opportunity to interact with faculty and staff in an informal environment, learning about their attitudes, values and behaviors and potentially deepening their sense of community.
      • Cleland J
      • Walker KG
      • Gale M
      • et al.
      Simulation-based education: understanding the socio-cultural complexity of a surgical training ‘boot camp’.
      Therefore, boot camps could potentially play a role in alleviating the stress and anxiety of early internship and even prevent burnout. This and other areas would benefit from more in-depth study.
      Thought should also be given to whether a 1- or 2-week course in medical school could reasonably affect performance in internship without additional preparation throughout the fourth year of medical school and during early internship. Although pediatric boot camps may be an important element to prepare students for the next stage of training, it seems essential for other elements of the fourth year to provide additional direct focus on residency preparedness, either as part of subinternships, a longitudinal curriculum, or other programs. As pediatric boot camps are more commonly becoming integrated into medical school curricula, institutions will need to decide whether the time- and faculty-intensive requirements of medical school boot camps are worth the institutional investment given the current limited evidence for improved intern performance. Leaders of pediatric boot camps have an opportunity to work with leaders of both medical schools and residency programs to redefine pediatric boot camps and residency preparation courses in order to improve the transition to pediatric residency by providing support for students with knowledge or skills gaps, enhancement of professional development, and effective communication with residency program directors.
      The fourth year of medical school continues to be an ill-defined phase of training, and we still do not fully understand the elements of curricular programming during this time that contribute to optimal intern performance. Future studies of medical school pediatric boot camps will help to elucidate the ideal curricular approach during this phase of training.

      Acknowledgments

      Financial statement: This study was supported by a grant from the Council on Medical Student Education in Pediatrics (COMSEP). Alan Schwartz serves as Director of the APPD LEARN research network through a contract from APPD to University of Illinois at Chicago. Beth King, MPP, APPD LEARN project manager, assisted sites with study onboarding and data collection. This work was part of her paid employment for the APPD LEARN Research Network. COMSEP had no role in study design, data collection, analysis/interpretation of data, or in the writing or decision to submit this manuscript.
      Group information: The Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network/Council on Pediatric Medical Student Education in Pediatrics (APPD LEARN/COMSEP) Boot Camp Study Group includes H Anwar, MD (Virginia Commonwealth University), M Bar-On, MD (University of Nevada Las Vegas), S Ben-Zion, MD (Children's Hospital Medical Center of Akron), R Burns, MD (Seattle Children's Hospital), M Carter, MD (Inova Children's Hospital), EP Devon, MD (Children's Hospital of Philadelphia), K Donnelly, MD (Inova Children's Hospital), M Fakaosita, MD (University of Hawaii), E Frank, MD (Rainbow Babies and Children's Hospital), J Goldstein, MD (Rainbow Babies and Children's Hospital), A Hartke, MD PhD (University of South Carolina School of Medicine Greenville), M Hormann, MD (University of Texas at Houston), O Kas-Osoka, MD, MEd (University of Nevada Las Vegas), J Kern, MD (Children's National Medical Center), B King, MPP (APPD LEARN, McLean Virginia), J Kramer, MD (Rush University Medical Center), S Lavoie, MD (Virginia Commonwealth University), S-T Li, MD, MPH (University of California Davis Health System), S Pishko, MD (University of Tennessee Health Science Center), WV Raszka, MD (University of Vermont Children's Hospital), M Rideout, MD (University of Vermont Children's Hospital), J Rinehart, MD (University of Vermont Children's Hospital), J Ronan, MD (Children's Hospital of Philadelphia), C Rusciolelli, MD (Rush University Medical Center), R Scherzer, MD (Nationwide Children's Hospital), A Schwartz, PhD (University of Illinois at Chicago and APPD LEARN, McLean, Virginia), K Sieplinga, MD (Helen DeVos Children's Hospital), C Skurkis, MD (Connecticut Children's Hospital), J Triemstra, MD (Helen DeVos Children's Hospital), S Unti, MD (Ann and Robert H. Lurie Children's Hospital of Chicago).

      Appendix. Supplementary Data

      References

        • Lyss-Lerman P
        • Teherani A
        • Aagaard E
        • et al.
        What training is needed in the fourth year of medical school? Views of residency program directors.
        Acad Med. 2009; 84: 823-829
        • Walling A
        • Merando A
        The fourth year of medical education. A literature review.
        Acad Med. 2010; 85: 1698-1704
        • Goldfarb S
        • Morrison G.
        The 3-year medical school–change or shortchange?.
        N Engl J Med. 2013; 369: 1087-1089
        • Kanter SL.
        How to win an argument about the senior year of medical school.
        Acad Med. 2009; 84: 815-816
        • Benson N
        • Stickle T
        • Raszka W.
        Going “Fourth” from medical school: fourth-year medical students’ perspectives on the fourth year of medical school.
        Acad Med. 2015; 90: 1386-1393
        • Teo AR
        • Harleman E
        • O'Sullivan P
        • et al.
        The key role of a transition course in preparing medical students for internship.
        Acad Med. 2011; 86: 860-865
        • Reddy ST
        • Chao J
        • Carter JL
        • et al.
        Alliance for clinical education perspective paper: recommendations for redesigning the “final year” of medical school.
        Teach Learn Med. 2014; 26: 420-427
        • American Board of Surgery
        Statement on surgical pre-residency preparatory courses.
        World J Surg. 2014; 38: 2743-2745
        • Neylan CJ
        • Nelson EF
        • Dumon KR
        • et al.
        Medical school surgical boot camps: a systematic review.
        J Surg Educ. 2017; 74: 384-389
        • Blackmore C
        • Austin J
        • Lopushinsky S
        • et al.
        Effects of postgraduate medical education "boot camps" on clinical skills, knowledge, and confidence: a meta-analysis.
        J Grad Med Educ. 2014; 6: 643-652
        • Burns R
        • Adler M
        • Mangold K
        • et al.
        Brief boot camp for 4th-year medical students entering into pediatric and family medicine residencies.
        Cureus. 2016; 8: e488
        • Green C
        • Vaughn C
        • Wyles S
        • et al.
        Evaluation of a surgery-based adjunct course for senior medical students entering surgical residencies.
        J Surg Educ. 2016; 73: 631-638
        • Minter R
        • Amos K
        • Bentz M
        • et al.
        Transition to surgical residency: a multi-institutional study of perceived intern preparedness and the effect of a formal residency preparatory course in the fourth year of medical school.
        Acad Med. 2015; 90: 1116-1124
        • Okusanya O
        • Kornfield Z
        • Reinke C
        • et al.
        The effect and durability of a pregraduation boot cAMP on the confidence of senior medical student entering surgical residencies.
        J Surg Educ. 2012; 69: 536-543
        • Bontempo L
        • Frayha N
        • Dittmar P.
        The internship preparation camp at the University of Maryland.
        Postgrad Med J. 2017; 93: 8-14
        • Pete Devon E
        • Tenney-Soeiro R
        • Ronan J
        • et al.
        A pediatric preintern boot camp: program development and evaluation informed by a conceptual framework.
        Acad Ped. 2019; 19: 165-169
        • Antonoff M
        • Swanson J
        • Green C
        • et al.
        The significant impact of a competency-based preparatory course for senior medical students entering surgical residency.
        Acad Med. 2012; 87: 308-319
        • Costich M
        • Finkel M
        • Friedman S
        • et al.
        Transition-to-residency: pilot innovative, online case-based curriculum for medical students preparing for pediatric internships.
        Med Educ Online. 2021; 261892569
      1. General Pediatrics Reported Milestones. Accreditation Council for Graduate Medical Education. 2015. Available at: https://www.acgme.org/globalassets/pdfs/milestones/crosswalkpediatricsreportingmilestones.pdf. Accessed February 22, 2022.

        • Englander R
        • Cameron T
        • Ballard A
        • et al.
        Toward a common taxonomy of competency domains for the health professions and competencies for physicians.
        Acad Med. 2013; 88: 1088-1094
        • Schwartz A
        • King B
        • Mink R
        • et al.
        The emergence and spread of practice-based medical education research networks.
        Acad Med. 2020; 95: S12-S13
        • Hartke A
        • Devon EP
        • Burns R
        • et al.
        Building a boot camp: pediatric residency preparatory course design workshop and took kit.
        MedEdPORTAL. 2019; 15: 10860
        • Hu K
        • Hicks PJ
        • Margolis M
        • et al.
        Reported Pediatrics Milestones (mostly) measure program, not learner performance.
        Acad Med. 2020; 95: S89-S94
      2. Obeso V Brown D Phillipi C Core Entrustable Professional Activities for Entering Residency: Toolkits for the 13 Core EPA’s - Abridged. Association of American Medical Colleges, 2017 (Available at:) (. Accessed February 22, 2022)
        • Lovell E
        • Mejicano G
        • co-chairs
        The Coalition for Physician Accountability’s Undergraduate Medical Education-Graduate Medical Education Review Committee (UGRC): Recommendations for Comprehensive Improvement of the UME-GME Transition.
        Coalition for Physician Accountability, 2021 (Available at:) (. Accessed February 22, 2022)
        • Cleland J
        • Walker KG
        • Gale M
        • et al.
        Simulation-based education: understanding the socio-cultural complexity of a surgical training ‘boot camp’.
        Med Educ. 2016; 50: 829-841https://doi.org/10.1111/medu.13064