Measuring Training Disruptions Using an Informatics Based Tool

Published:March 16, 2022DOI:



      Training disruptions, such as planned curricular adjustments or unplanned global pandemics, impact residency training in ways that are difficult to quantify. Informatics-based medical education tools can help measure these impacts. We tested the ability of a software platform driven by electronic health record data to quantify anticipated changes in trainee clinical experiences during the COVID-19 pandemic.


      We previously developed and validated the Trainee Individualized Learning System (TRAILS) to identify pediatric resident clinical experiences (i.e. shifts, resident provider-patient interactions (rPPIs), and diagnoses). We used TRAILS to perform a year-over-year analysis comparing pediatrics residents at a large academic children's hospital during March 15–June 15 in 2018 (Control #1), 2019 (Control #2), and 2020 (Exposure).


      Residents in the exposure cohort had fewer shifts than those in both control cohorts (P < .05). rPPIs decreased an average of 43% across all PGY levels, with interns experiencing a 78% decrease in Continuity Clinic. Patient continuity decreased from 23% to 11%. rPPIs with common clinic and emergency department diagnoses decreased substantially during the exposure period.


      Informatics tools like TRAILS may help program directors understand the impact of training disruptions on resident clinical experiences and target interventions to learners’ needs and development.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Academic Pediatrics
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Lieu TA
        • Forrest CB
        • Blum NJ
        • et al.
        Effects of a night-float system on resident activities and parent satisfaction.
        Am J Dis Children. 1992; 146: 307-310
        • Osborn R
        • Bullis E
        • Fenick AM
        • et al.
        X + Y scheduling in pediatric residency: continuity, handoffs, and trainee experience.
        Acad Pediatr. 2019; 19: 489-494
        • Geanacopoulos AT
        • Sundheim KM
        • Greco KF
        • et al.
        Pediatric intern clinical exposure during the COVID-19 pandemic.
        Hosp Pediatr. 2021; 11: e106-e110
        • Chiel L
        • Winthrop Z
        • Winn AS
        The COVID-19 pandemic and pediatric graduate medical education.
        Pediatrics. 2020; 146e20201057
        • Edigin E
        • Eseaton PO
        • Shaka H
        • et al.
        Impact of COVID-19 pandemic on medical postgraduate training in the United States.
        Med Educ Online. 2020; 251774318
        • Cadish LA
        • Fung V
        • Lane FL
        • et al.
        Surgical case logging habits and attitudes: a multispecialty survey of residents.
        J Surg Educ. 2016; 73: 474-481
        • Gill DJ
        • Freeman WD
        • Thoresen P
        • et al.
        The neurology resident case log: a national survey of neurology residents.
        Neurology. 2007; 68: E32-E33
        • Arora VM
        Harnessing the power of big data to improve graduate medical education: big idea or bust?.
        Acad Med. 2018; 93: 833-834
      1. Accreditation Council for Graduate Medical Education: Common Program Requirements [Internet]. 2017. Available at: Accessed March 11, 2022.

        • Mai MV
        • Orenstein EW
        • Manning JD
        • et al.
        Attributing patients to pediatric residents using electronic health record features augmented with audit logs.
        Appl Clin Inform. 2020; 11: 442-451
        • Dziorny AC
        • Orenstein
        • Evan W
        • et al.
        Automatic detection of front-line clinician hospital shifts: a novel use of EHR timestamp data.
        Appl Clin Inform. 2019; 10: 28-37
        • Downing SM
        • Haladyna TM
        Validity and its threats.
        Assessment in Health Professions Education. Routledge Taylor and Francis Group, New York, NY2009
        • Chaiyachati BH
        • Agawu A
        • Zorc JJ
        • et al.
        Trends in pediatric emergency department utilization after institution of Coronavirus disease-19 mandatory social distancing.
        J Pediatr. 2020; 226 (274-277.e1)
        • Patel SY
        • Mehrotra A
        • Huskamp HA
        • et al.
        Trends in outpatient care delivery and telemedicine during the COVID-19 pandemic in the US.
        JAMA Intern Med. 2021; 181: 388-391
        • Clinical Classifications Software Refined (CCSR)
        For ICD-10-CM Diagnoses, v2021.1 [Internet].
        Agency for Healthcare Research and Quality, 2020 (Healthcare Cost and Utilization Project (HCUP)). Available at: (Accessed April 26, 2022)
        • Hartnett KP
        • Kite-Powell A
        • DeVies J
        • et al.
        Impact of the COVID-19 pandemic on emergency department visits — United States, January 1, 2019–May 30, 2020.
        MMWR Morb Mortal Wkly Rep. 2020; 69: 699-704
        • Wilder JL
        • Parsons CR
        • Growdon AS
        • et al.
        Pediatric hospitalizations during the COVID-19 pandemic.
        Pediatrics. 2020; 146e2020005983
        • Hilburg R
        • Patel N
        • Ambruso S
        • et al.
        Medical education during the Coronavirus disease-2019 pandemic: learning from a distance.
        Adv Chronic Kidney Dis. 2020; 27: 412-417
        • Woods SK
        • Turner DA
        Training Disruptions Due to COVID-19 for Residents and Fellows Graduating in 2021.
        American Board of Pediatrics, 2020 (Available at:)