Advertisement

Development, Implementation, and Assessment of the Intensive Clinical Orientation for Residents (ICOR) Curriculum: A Pilot Intervention to Improve Intern Clinical Preparedness

Published:November 24, 2017DOI:https://doi.org/10.1016/j.acap.2017.11.004
      Each June, medical students transition to residency as newly minted doctors. Interns assume far more responsibility than they had during medical school
      • Raymond M.R.
      • Mee J.
      • King A.
      • et al.
      What new residents do during their initial months of training.
      while concurrently learning new hospital systems. Medical educators and faculty also face challenges, as the knowledge base and skill sets of new interns vary
      • Gollehon N.
      • Stansfield B.
      • Gruppen L.
      • et al.
      Assessing residents' competency at baseline: how much does the medical school matter?.
      and their supervisors need to ensure safe and efficient care for patients. Across specialties, recent literature has demonstrated that many program directors perceive new residents to be underprepared for residency.
      • Lyss-Lerman P.
      • Teherani A.
      • Aagaard E.
      • et al.
      What training is needed in the fourth year of medical school? Views of residency program directors.
      Accordingly, in 2014, the Association of American Medical Colleges (AAMC) published a list of core entrustable professional activities (EPAs) that all graduating medical students are expected to be able to perform on day 1 of residency without direct supervision.
      • Association of American Medical Colleges
      Core entrustable professional activities for entering residency.
      By more clearly defining expectations and by asking medical schools to pilot curricula to address these EPAs, the AAMC hopes to improve the preparation of medical students entering residency training.
      Although some medical schools have started to offer experiences to prepare fourth-year medical students for the clinical realities of internship,
      • Blackmore C.
      • Austin J.
      • Lopushinsky S.R.
      • et al.
      Effects of postgraduate medical education “boot camps” on clinical skills, knowledge, and confidence: a meta-analysis.
      • Bontempo L.J.
      • Frayha N.
      • Dittmar P.C.
      The internship preparation camp at the University of Maryland.
      • Burns R.
      • Adler M.
      • Mangold K.
      • et al.
      A brief boot camp for 4th-year medical students entering into pediatric and family medicine residencies.
      • Laack T.A.
      • Newman J.S.
      • Goyal D.G.
      • et al.
      A 1-week simulated internship course helps prepare medical students for transition to residency.
      • Minha S.
      • Shefet D.
      • Sagi D.
      • et al.
      “See one, sim one, do one”—a national pre-internship boot-camp to ensure a safer “student to doctor” transition.
      • Wayne D.B.
      • Cohen E.R.
      • Singer B.D.
      • et al.
      Progress toward improving medical school graduates' skills via a “boot camp” curriculum.
      these experiences mostly occur outside of trainees' future clinical environment. This may be problematic because achieving competency in many of the EPAs is context dependent, contingent on an intern's ability to navigate hospital systems and integrate into the clinical working environment. To date, we are unaware of a robust curriculum based in interns' new environments that targets these EPAs. Some programs have begun to provide new physicians in training with boot camps inclusive of simulated clinical experiences or classroom-based experiences, which have been shown to be beneficial in preliminary studies.
      • Blackmore C.
      • Austin J.
      • Lopushinsky S.R.
      • et al.
      Effects of postgraduate medical education “boot camps” on clinical skills, knowledge, and confidence: a meta-analysis.
      • Krajewski A.
      • Filippa D.
      • Staff I.
      • et al.
      Implementation of an intern boot camp curriculum to address clinical competencies under the new Accreditation Council for Graduate Medical Education supervision requirements and duty hour restrictions.
      • Cohen E.R.
      • Barsuk J.H.
      • Moazed F.
      • et al.
      Making July safer: simulation-based mastery learning during intern boot camp.
      • Fernandez G.L.
      • Page D.W.
      • Coe N.P.
      • et al.
      Boot camp: educational outcomes after 4 successive years of preparatory simulation-based training at onset of internship.
      • Min A.A.
      • Stoneking L.R.
      • Grall K.H.
      • et al.
      Implementation of the introductory clinician development series: an optional boot camp for emergency medicine interns.
      • Seeley M.A.
      • Kazarian E.
      • King B.
      • et al.
      Core concepts: orthopedic intern curriculum boot camp.
      However, while these experiences provide simulated procedural training or classroom-based teaching, training in actual clinical contexts with real patients is typically lacking. Therefore, most interns assume patient care responsibilities without any hands-on clinical training in their new environments. Given that many EPAs are specific to the context of a clinical experience, we introduced a novel training experience, based in an actual clinical setting at an intern's receiving (residency) rather than sending (medical school) institution, to better prepare trainees for starting internship.

      Approach

      To develop this educational intervention, we followed Kern's 6-step approach
      • Kern D.E.
      • Thomas P.A.
      • Hughes M.T.
      Curriculum Development for Medical Education: A Six-Step Approach.
      to guide the development and implementation of our program, Intensive Clinical Orientation for Residents (ICOR) (Table 1).
      Table 1Application of Kern and Colleagues' 6 Steps of Curriculum Development to ICOR Curriculum
      • Kern D.E.
      • Thomas P.A.
      • Hughes M.T.
      Curriculum Development for Medical Education: A Six-Step Approach.
      Curricular Development StepAssociated ICOR Curricular Development Activities
      1. Problem identification and needs assessment
      • Problem of lack of consistent clinical preparation for starting training programs identified by:
        • Reviewing the AAMC EPAs for graduating medical students.
        • Convening an Institutional Education Strategic Task Force.
      • General needs assessment performed by literature review.
      • Current construct of starting intern year compared to theoretical ideal construct.
      2. Targeted needs assessment
      • Targeted needs assessment performed by electronically surveying current residents and core attending faculty, asking them to rate the importance of teaching a series of EPAs in the ICOR, including as options both the AAMC's core EPAs and others pertinent to our learning environment.
      • Non-AAMC EPAs developed with input from a purposeful sample including chief residents, hospitalists, and medical educators.
      3. Goals and objectives
      • Overall goal to improve confidence and preparedness when starting intern year developed by key stakeholders.
      • For specific objectives of our curriculum, EPAs included that more than 50% of residents or attending faculty thought were “very important” to teach in the ICOR.
      • Excluded EPA, to recognize a patient that requires urgent or emergent care and initiate evaluation and management, as these events are relatively rare, making it hard to guarantee hands-on experience during a clinical boot camp.
      4. Educational strategies
      • Multiple educational strategies considered and included to target different types of learners as well as higher-level objectives:
        • Real-life clinical experience.
        • Environmental interventions (decreased patient load and increased supervision).
        • Role modeling.
        • Feedback on performance.
        • Role-play.
        • Small-group learning.
        • Interactive lectures.
      5. Implementation
      • Supervising faculty and residents recruited and trained.
      • Intern and facilitator's guides developed.
      • Lectures developed and performed.
      • Curriculum piloted.
      6. Evaluation and feedback
      • Feedback on curriculum obtained.
      • Iterative changes made.
      • Formal study performed to compare interns who participated in boot camp with control group.
      AAMC indicates Association of American Medical Colleges; EPA, entrustable professional activities; and ICOR, Intensive Clinical Orientation for Residents.

       Curriculum

      The goal of the ICOR was to improve confidence and preparedness when starting intern year. On the basis of our needs assessment, we chose to focus the curriculum on a list of 9 EPAs—6 directly from the AAMC's EPAs, and 3 specific to our learning environment. The 6 AAMC EPAs included: 1) provide an oral presentation of a clinical encounter, 2) prioritize a differential diagnosis after a clinical encounter, 3) document a clinical encounter in the patient record, 4) give or receive a patient handover to transition care responsibility, 5) enter and discuss orders and prescriptions, and 6) collaborate as a member of an interprofessional team. The 3 non-AAMC EPAs included: 7) formulate an assessment of a patient that is modified throughout a patient's hospital course, 8) prioritize clinical tasks, and 9) call a consult, framing the call with a clinical question.
      The ICOR curriculum includes 2 main components: a hands-on clinical component and a classroom-based component. First and foremost, we wanted interns to learn in the environment in which they would practice—on an inpatient general pediatrics service. In this curriculum, new interns spent 3 days on the general pediatric service performing the usual functions performed in intern year, such as admitting patients, participating in handoffs, and entering orders. However, they did so with decreased patient volume and increased supervision, allowing for a more supported learning environment. In order to decrease patient volume, each new intern cared for one patient on the first day of the ICOR and up to 4 patients by the end of the 3 days. The remaining patients were cared for by outgoing interns. We increased our supervision by allowing outgoing interns to assist with supervision. We also incorporated one volunteer faculty member or chief resident per team to assist with timely feedback. Current residents role modeled challenging tasks of intern year, such as patient handovers and oral presentations, before the new interns performed these tasks themselves. Intensive feedback was incorporated into the ICOR, with new interns receiving feedback from both supervising residents and attending physicians on all the EPAs they were performing.
      Next, throughout the ICOR, we included twice-daily interactive workshops, utilizing lectures, small-group discussions, role-play, and problem-based learning. Example workshop topics include I-PASS handoffs
      • Starmer A.J.
      • O'Toole J.K.
      • Rosenbluth G.
      • et al.
      Development, implementation, and dissemination of the I-PASS handoff curriculum: a multi-site educational intervention to improve patient handoffs.
      and presenting on family-centered rounds. The full schedule is listed in Table 2. Therefore, for each EPA, the new interns went through an intentional sequence of skill development including observation in a clinical setting, workshop discussion, followed by practicing the EPA in a supervised fashion with immediate formative feedback.
      Table 2Intensive Clinical Orientation for Residents Schedule
      ActivityTimeDay 1
      Before starting the clinical curriculum, interns participated in 3 targeted workshops: I-PASS–Based Handoffs, Presenting on Family-Centered Rounds, and Developing an Assessment and Plan. On the morning of day 1, interns cared for one patient. Later that afternoon, they started caring for an additional 1 to 3 patients.
      Day 2Day 3
      Sign in6:30–7:30 amNew interns observe I-PASS
      • Starmer A.J.
      • O'Toole J.K.
      • Rosenbluth G.
      • et al.
      Development, implementation, and dissemination of the I-PASS handoff curriculum: a multi-site educational intervention to improve patient handoffs.
      handoffs and practice synthesis by receiver on 1 assigned patient.
      New interns participate in I-PASS handoffs on all patients and receive feedback.
      Prerounding7:30–8:00 amSupervising residents walk through strategies for prerounding on assigned patient with interns.New interns preround with direct supervision and receive feedback.
      Rounds8:00–10:00 amCurrent residents role model family-centered rounds for the majority of patients. New interns present their assigned patient and receive feedback on their presentation.New interns present all of their patients (2–4 each) during family-centered rounds and receive verbal and written feedback on their presentations.
      Working session10:00 am–12:00 pmTrainees are observed and guided when entering orders, calling consults, writing notes, and admitting/discharging patients. Interns receive verbal and written feedback on all of these tasks from attending physicians and supervising residents.
      Lecture/workshop12:00–1:00 pmDocumentationAdvanced lecture on Formulating an Assessment and PlanCalling a Consult
      Working session1:00–3:00 pmSame as above.
      Lecture/workshop3:00–3:45 pmInterdisciplinary CommunicationPrioritization of a To-Do ListEKG Basics; Common Antibiotics
      Working session3:45–5:30 pmSame as above.
      Sign out5:30–6:30 pmNew interns participate in I-PASS handoffs on all patient and receive verbal and written feedback.
      * Before starting the clinical curriculum, interns participated in 3 targeted workshops: I-PASS–Based Handoffs, Presenting on Family-Centered Rounds, and Developing an Assessment and Plan. On the morning of day 1, interns cared for one patient. Later that afternoon, they started caring for an additional 1 to 3 patients.

       Implementation

      The 3½-day ICOR was piloted for half of the incoming pediatrics interns in the Boston Combined Residency Program (n = 24) just before starting intern year in June 2015, at the end of intern orientation. Eighteen attending physicians and 27 supervising residents participated in the ICOR program. Intern and supervisor guides were developed, with a section of best practices for each EPA, and disseminated. All attending physician and resident supervisors were trained to give verbal and written feedback, and to model best practices by participating in 1 of 8 training sessions before the start of the ICOR.

       Program Evaluation

      The primary outcome of this study was intern perceived preparedness to start intern year. We evaluated this by randomizing half of our incoming interns to participate in the ICOR and half to participate in a separate pilot simulation exercise. The simulation exercise targeted the AAMC's EPA of recognizing a patient requiring urgent or emergent care and initiating evaluation and management, which was not addressed in the ICOR. A subset of each of these 2 groups started their internship working on inpatient wards, while other interns started in outpatient clinics and in other settings. We asked the subset who worked on the inpatient wards to retrospectively rate how prepared they felt to start intern year on a 10-point scale (1 = not at all, 5 = somewhat, 10 = extremely) at the end of their first rotation, then compared the responses of those inpatient interns who had been randomized to ICOR (exposed group) to the unexposed group.
      In addition, all ICOR participants, including those who did not start intern year on an inpatient rotation, completed a survey regarding their perceptions of the ICOR. First, they were asked to rate the ICOR's educational value on a 5-point Likert scale from 1 = very negative to 5 = very positive. They were asked to rate the statement, “I feel more confident starting intern year [because of the ICOR]” on a similar 5-point Likert scale from 1 = strong disagree to 5 = strongly agree. For the purposes of analysis, both of these scales were dichotomized into top 2 boxes versus all other boxes. Additionally, participants were asked whether the ICOR should be offered again the next year.
      To assess whether the program was meeting its goals, we asked a series of process-oriented questions. We asked participants to choose one patient for whom they cared during the ICOR and to reflect on whether they performed the EPAs listed in our learning objectives when caring for that patient, and whether they received feedback on these EPAs. The institutional review boards at Boston Children's Hospital and Boston Medical Center reviewed the protocol for evaluation and determined that it qualified as exempt.

      Outcomes

      Of our 48 interns, 24 participated in the ICOR and 24 participated in the simulation exercise instead. Only 22 of the 48 interns started intern year on an inpatient rotation and were included in the comparison study. These included 10 ICOR interns (exposed group) and 12 simulation interns (unexposed group). At the end of the first block of intern year, exposed interns, who had participated in the ICOR, retrospectively rated themselves as more prepared to start intern year (mean 7.0 out of 10, standard deviation 1.0, n = 10) compared to the unexposed group (mean 5.6, standard deviation 1.8; P = .0496, n = 11). Response rate for this survey was 95% (21 of 22).
      The response rate for the second survey was 58% (14 of 24). This included all 24 of the ICOR residents, including those who did not work in the inpatient setting initially and were therefore not included in the previous survey. In this survey, 100% of respondents thought the curriculum should be offered again and rated its educational value as very positive (71%) or positive (29%). Ninety-three percent of interns reported feeling more confident starting intern year because of the curriculum.
      New interns were asked to reflect on one patient for whom they cared during the ICOR. For 6 of the 9 EPAs, more than 90% of interns reported performing the EPA when taking care of their patient. The least frequently performed EPA was “calling a consult,” with only 50% of interns reporting performing that EPA. For 8 of 9 EPAs, the majority of residents (58–83%) reported receiving feedback on the EPA when performed.

      Discussion

      Interns felt better prepared and more confident to start intern year with institution of the ICOR, a novel orientation that took place in the interns' new clinical environment. During the ICOR, participants were able to practice and receive feedback on the majority of EPAs, underscoring the feasibility of targeting EPA-based learning objectives during a clinically based orientation. However, certain EPAs, such as “calling a consult,” were practiced less frequently, likely because this EPA occurs less frequently in daily practice than other EPAs, such as “writing orders and prescriptions.”
      The ICOR utilized many principles of adult learning theory.
      • Knowles M.
      The Modern Practice of Adult Education.
      For example, adult learners are more engaged and motivated to learn when they understand the benefits of learning new material. Situating the ICOR directly before the beginning of intern year provided interns with an immediate understanding of the benefits of the curriculum and inherent motivation to maximize learning from the curriculum. Medical school programs that provide clinical orientations months before intern year begins may not have this advantage. Adult learners are more engaged when they can apply their past experiences to their current learning (eg, interns' experiences in medical school). The ICOR allows interns to build on their already developed skill set by practicing those same skills in real-life scenarios and receiving feedback tailored to their prior experiences. Finally, adult learning theory also states that adult learners benefit from immediate application of their learning. This was demonstrated by the ICOR's occurring in the interns' actual training environment. Application of the ICOR in an actual clinical setting differentiates it most from other lecture or simulation-based preparatory curricula or boot camps.
      There were several limitations to the ICOR program and its study. We assessed self-reported preparedness and did not include objective measures of preparedness or competence. Despite the potential inaccuracies of self-assessment, improved feelings of preparedness and confidence may help with the common sentiment of being overwhelmed at the beginning of intern year. Because the ICOR has now become our standard practice in the wake of its successful pilot, our objective is to continue to study this intervention and determine whether it leads to improved objective preparedness and competency and earlier entrustment of professional activities. Another limitation is that the unexposed interns, who did not participate in the ICOR, received a simulation curriculum. It is possible that this experience also increased the unexposed group's perceived preparedness to start intern year, leading to an underestimation of the benefit of the ICOR.
      The ICOR was successful in improving the perceived preparedness and confidence of new interns. We believe that the ability to learn these skills in a relevant clinical setting with environmental modifications is a novel approach and is integral to its success. Since the initial implementation, we have scaled the program to include nongeneral pediatrics services, such as an inpatient pulmonary service. We anticipate that this model could be adopted within other training programs in pediatrics and other disciplines, and that it would supplement medical school and simulation-based boot camps that have already been shown to be successful.

      Acknowledgment

      Supported in part by a Boston Children's Hospital Residency Funding Grant, the Fred Lovejoy Resident Research and Education Award.

      References

        • Raymond M.R.
        • Mee J.
        • King A.
        • et al.
        What new residents do during their initial months of training.
        Acad Med. 2011; 86: S59-S62
        • Gollehon N.
        • Stansfield B.
        • Gruppen L.
        • et al.
        Assessing residents' competency at baseline: how much does the medical school matter?.
        Acad Med. 2016; 91: S12-S13
        • Lyss-Lerman P.
        • Teherani A.
        • Aagaard E.
        • et al.
        What training is needed in the fourth year of medical school? Views of residency program directors.
        Acad Med. 2009; 84: 823-829
        • Association of American Medical Colleges
        Core entrustable professional activities for entering residency.
        (Available at:)
        https://www.mededportal.org/icollaborative/resource/887
        Date: 2013
        Date accessed: February 1, 2017
        • Blackmore C.
        • Austin J.
        • Lopushinsky S.R.
        • et al.
        Effects of postgraduate medical education “boot camps” on clinical skills, knowledge, and confidence: a meta-analysis.
        J Grad Med Educ. 2014; 6: 643-652
        • Bontempo L.J.
        • Frayha N.
        • Dittmar P.C.
        The internship preparation camp at the University of Maryland.
        Postgrad Med J. 2017; 93: 8-14
        • Burns R.
        • Adler M.
        • Mangold K.
        • et al.
        A brief boot camp for 4th-year medical students entering into pediatric and family medicine residencies.
        Cureus. 2016; 8: e488
        • Laack T.A.
        • Newman J.S.
        • Goyal D.G.
        • et al.
        A 1-week simulated internship course helps prepare medical students for transition to residency.
        Simul Healthc. 2010; 5: 127-132
        • Minha S.
        • Shefet D.
        • Sagi D.
        • et al.
        “See one, sim one, do one”—a national pre-internship boot-camp to ensure a safer “student to doctor” transition.
        PLoS ONE. 2016; 11 (e0150122)
        • Wayne D.B.
        • Cohen E.R.
        • Singer B.D.
        • et al.
        Progress toward improving medical school graduates' skills via a “boot camp” curriculum.
        Simul Healthc. 2014; 9: 33-39
        • Krajewski A.
        • Filippa D.
        • Staff I.
        • et al.
        Implementation of an intern boot camp curriculum to address clinical competencies under the new Accreditation Council for Graduate Medical Education supervision requirements and duty hour restrictions.
        JAMA Surg. 2013; 148: 727-732
        • Cohen E.R.
        • Barsuk J.H.
        • Moazed F.
        • et al.
        Making July safer: simulation-based mastery learning during intern boot camp.
        Acad Med. 2013; 88: 233-239
        • Fernandez G.L.
        • Page D.W.
        • Coe N.P.
        • et al.
        Boot camp: educational outcomes after 4 successive years of preparatory simulation-based training at onset of internship.
        J Surg Educ. 2012; 69: 242-248
        • Min A.A.
        • Stoneking L.R.
        • Grall K.H.
        • et al.
        Implementation of the introductory clinician development series: an optional boot camp for emergency medicine interns.
        Adv Med Educ Pract. 2014; 5: 275-279
        • Seeley M.A.
        • Kazarian E.
        • King B.
        • et al.
        Core concepts: orthopedic intern curriculum boot camp.
        Orthopedics. 2016; 39: e62-e67
        • Kern D.E.
        • Thomas P.A.
        • Hughes M.T.
        Curriculum Development for Medical Education: A Six-Step Approach.
        Johns Hopkins University Press, Baltimore, Md1998
        • Starmer A.J.
        • O'Toole J.K.
        • Rosenbluth G.
        • et al.
        Development, implementation, and dissemination of the I-PASS handoff curriculum: a multi-site educational intervention to improve patient handoffs.
        Acad Med. 2014; 89: 876-884
        • Knowles M.
        The Modern Practice of Adult Education.
        Prentice Hall, Englewood Cliffs, NJ1980