If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Address correspondence to Megan E. McCabe, MD, Department of Pediatrics/Division of Pediatric Critical Care Medicine, The Children's Hospital at Montefiore/Albert Einstein College of Medicine, 3411 Wayne Ave, Bronx, NY, 10467
To create and validate a checklist for high-quality documentation and pilot a multi-modal, immersive educational module across multiple institutions. We hypothesized that this module would improve knowledge, skills, and attitudes in medical documentation.
Module design was grounded in an established curriculum design framework. We conducted the study across 12 pediatric critical care fellowship programs between September 2017 and January 2018. Workshops were allotted 90 minutes for completion. We utilized a pre-/post- study design to determine the workshop's impact. Changes in knowledge were assessed through pre and post testing. Changes in skills were evaluated with a validated checklist for inclusion of key documentation elements. Changes in attitudes were determined through learner self-assessment
83 of 138 eligible fellows (60%) started the module and 62 of 83 (75%) completed data sets for analysis. Immediate post-testing demonstrated modest statistically significant improvement in knowledge, skills, and attitudes. The workshop was easily disseminated and deployed
This study demonstrates that a multi-modal educational intervention can lead to improvement in medical documentation knowledge, skills, and attitudes in a cohort of PCCM fellows and be easily disseminated for use by other specialties and types of clinicians.
A 90-minute workshop using a validated documentation checklist increased pediatric critical care fellow knowledge, attitudes and skill in documenting unexpected events and family meetings. This workshop is now available for adaptation to other specialties for trainee education in this neglected area.
High-quality documentation is of critical importance to patient care, multidisciplinary communication, and financial reimbursement; it also has implications for individual provider and institutional medico-legal risk.
The Education in Pediatric Intensive Care (EPIC) Investigators, in collaboration with colleagues from institutional legal and risk management departments, created a multimodal interactive curricular module using didactic, immersive, and peer-facilitated techniques to improve learners’ knowledge of essential elements for medical documentation. We hypothesized that pediatric critical care medicine (PCCM) fellows who participated in this module would demonstrate improved knowledge, skills, and attitudes/behaviors in documentation of clinically significant events and discussions of unanticipated adverse clinical events, medical errors, and goals of care. We piloted this module with the intent of dissemination to other training programs.
Curriculum Development and Assessment
Kern's 6-step framework for curriculum development
The EPIC Investigators, an experienced group of educators and researchers in PCCM, identified the need for improved rigor and processes for teaching medical documentation to trainees. A literature search confirmed the prevalence and impact of poor documentation across specialties and in a wide range of settings.
have generally been single-center studies focused on medical students and residents. This finding demonstrates a gap in current research in terms of documentation practices for advanced trainees (ie, fellows) and the need for implementation of a multisite intervention to enhance documentation practices.
Targeted Needs Assessment
PCCM Fellowship Program Directors and Risk Management professionals from 9 institutions participated in iterative exploratory interviews to broadly identify high-priority documentation needs. There was overall consensus that targeted needs for improved documentation included 4 key clinical events: 1) disclosure of an unanticipated adverse event; 2) disclosure of a medical error; 3) an “Event Note” for a significant change in clinical status; and 4) summation of discussions regarding goals of care.
Curricular Goals and Objectives
Goal: Through an interactive learning module, PCCM fellows will understand and apply the elements of high-quality clinical and medico-legal documentation.
Learner Focused Objectives
By the conclusion of the curricular module, learners would: 1) define the elements of high-quality documentation in the medical record, 2) discuss the importance of quality documentation and 3) demonstrate application of these skills to note writing of potentially high impact records, including clinically significant events and discussions of goals of care, unanticipated adverse clinical events, and medical errors with families/patients.
Module content was developed by experienced pediatric intensivists in collaboration with institutional legal and risk management colleagues. The module was designed to enhance knowledge, skills, and attitudes relevant to medical documentation.
A checklist was designed to standardize the rating of clinical documentation for the 4 scenarios identified in the targeted needs assessment. As suggested by Cook and Downing, validity evidence for the checklist was obtained in several areas. Content validity was supported by the incorporation of evidence for best practice in medical documentation identified in the literature search,
as well as the input and iterative revisions provided by experienced educators in pediatric critical care medicine and hospital risk management. Evidence for response process of the checklist was improved by pilot testing the use of the checklist with 7 experienced physician educators and 10 trainees and collecting feedback for clarification and improvement of the checklist items as well as ease of completion. Evidence supporting acceptable internal structure of the instrument included calculating Cronbach's alpha and the assessment of interrater reliability. Cronbach's alpha was 0.851, total scores ranged from 6-13. Interrater reliability for the checklist tool was assessed by calculation of intraclass correlation coefficient (ICC). The ICC for the total score was 0.85, 95% confidence interval (0.59, 0.97) using a 2-way random effects model.
Two written vignettes describing unplanned adverse events and 2 simulated family meeting videos were created by experienced medical educators and critically reviewed by all members of the study team. All aspects of the module were tested and revised, as needed, prior to implementation. All members of the study team approved the final version of the curriculum before it was distributed. (Appendix A)
Ericsson's conceptual framework of deliberate practice
served as the foundation for the educational intervention. The iterative process of learning, application, feedback, and refinement fits well with the acquisition and application of best practices in documentation.
The module was designed as an instructor-facilitated small-group interactive workshop. Required materials included laptops or paper/pencil for each learner, the instructor's guide, and accompanying didactic slides with videos embedded.
To start the workshop, learners were given a written clinical vignette of an unplanned adverse event in the pediatric intensive care unit (PICU) and composed a clinical note as they normally would within the patient's chart. The learners then reviewed their notes with peers using targeted discussion prompts provided by the facilitator. A facilitator started the didactic session, which outlined key principles of high-quality documentation. A second adverse event written vignette was presented for the learners to document and review with peers using the same discussion prompts. A video-recorded simulated family meeting was then presented. Learners again wrote notes and reviewed each one with their peers. Further information specific to documenting family meetings was presented and a second video-recorded simulated family meeting was shown. Learners again composed a note and reviewed with their peers to facilitate skill development and shared insights and reflections in group feedback. Workshop Agenda shown in Appendix B.
Individual elements of the curriculum module were piloted with small cohorts of graduating pediatric critical care fellows who did not contribute to the final data. The didactic teaching and facilitation of the interactive portions of the module were piloted with advanced pediatric educators as a workshop at the Association of Pediatric Program Directors Annual Spring Meeting in 2015.
Study sites were recruited by emails to program directors participating in the EPIC Investigators collaborative and through discussion during the group's twice monthly conference calls. The program directors agreed to participate by running the module during a regularly scheduled teaching time in their program and complete IRB approval as required by their institution.
The curriculum was distributed to 12 PCCM training programs in 2017 to 2018, comprising 18% of PCCM programs at the time of the study. Ninety-minutes were allotted for completion of the workshop. Knowledge, skills, and attitudes of participants were assessed pre- and postmodule. Knowledge of best practices in medical documentation of the 4 selected scenarios was evaluated through a 4-item multiple-choice test. (Appendix C) Participant skills were assessed by an independent evaluator who reviewed and scored all submitted notes using the 14-item checklist. (Appendix D) The evaluator (D.B.) was not a member of the training programs or project development group. The evaluator was blinded to whether notes were written in the pre or post module and any information about the author of the note. Participant attitudes were evaluated using a 5-point Likert scale to answer 2 questions regarding perceived importance and self-assessed skill in medical documentation.
All data analysis was conducted with SPSS 27.0 (IBM SPSS Statistics, Armonk, NY). Descriptive statistics including medians and counts/frequencies were reported for the variables of interest. One-way ANNOVA and paired samples t tests were conducted to measure the effectiveness of the curriculum in knowledge, attitudes, and skills. Data are mean (SD) unless otherwise indicated.
IRB approval was obtained at Albert Einstein College of Medicine for both the module development process and assessment. Additional IRB approvals for each site were obtained during the assessment phase.
Eighty-three PCCM fellows out of 138 (60%) across the 12 sites attended the workshop at their site. Sixty-two (75%) generated complete data sets for analysis. Seven individuals either did not consent to participate or did not provide demographic information and data. Fourteen individuals completed the demographics but not all components of the workshop. Demographics of the participants who did and did not complete the workshop are shown in Table 1. The majority (63%) of participants reported no prior medical documentation training. There were no significant differences in any of the measured outcomes when evaluated by PGY year.
Documentation knowledge improved from preworkshop to postworkshop. Attitudes regarding the importance of documentation improved significantly as did self-assessment in skill (Figure).
Participant skill in documentation of an unexpected adverse event increased from a mean premodule note score of 8.6/14 ± 1.3 to 10/14 ± 1.5 conditions t(61), -6.6 P < .001. Participant skills in documentation of a family meeting also improved (premodule note scores 10.4/13 ± 1.0); postmodule note scores 10.8/13 ± 1.5; conditions t(61), -3.1 P < .003) (Figure 1).
With regard to the specific elements of documentation highlighted in the checklist, 4 items improved markedly and reached statistical significance in the postmodule notes by all participants, (P < .05). These were: time frame of the events described (11%–100% included), participants identified (21%–100%), key personnel notified and attending physician named (4%–100%) and inclusion of a follow up plan (23%–100%). Other items demonstrating significant improvement were: patient/family notification (31%–50%) and patient/family response described (0%–52%). The remainder of the required elements for key documentation were present at high levels in the baseline notes. These included: patient status prior to event, sequence of events, lack of inappropriate notations, lack of Joint Commission “Do Not Use” and nonstandard abbreviations, use of objective language and interpretations based on fact. One item: “justification/decision making clearly outlined” declined from the baseline to the final note (61%–53%) (Table 2).
In this multicenter single specialty pilot, we were able to develop and test a curriculum in medical documentation that can be readily adapted for other clinical specialties to teach relevant, high impact high quality documentation. Building upon prior published work on teaching medical documentation targeted primarily towards medical students and residents, this curricular module was able to engage advanced learners and effect improvement in knowledge, skills, and attitudes.
Given the vitally important role of medical documentation to provide safe and effective patient care and the impact of the 21st Century Cures Act on real-time documentation access, it is imperative that clinicians have an understanding of documentation best practices, including who should document, what should be documented, and when documentation should occur.
It is equally essential to have proven effective methods for teaching effective documentation. Our project has generated an approach that is widely applicable to graduate medical trainees across many disciplines. While our specific learner population was pediatric critical care medicine trainees for this study, we have also piloted the workshop with general pediatricians at a national meeting and found it easily adaptable to both high acuity inpatient and low-acuity outpatient settings.
The majority (61%) of participants in this study reported no prior training in medical documentation despite their combined years of undergraduate and prefellowship graduate medical education. This is reflected in our finding that attitudes, knowledge and skills were not different based on level of training preintervention. As medical education continues to transition to a competency-based model of training, identifying the skills essential to unsupervised practice is a foundational task. Effective medical documentation is a skill that spans all specialties and disciplines. The Association of American Medical Colleges identifies medical documentation as a core Entrustable Professional Activity (EPA) for students to be skilled at prior to entering residency training.
These examples highlight the central role that documentation plays in patient care and as a professional activity. It is incumbent upon training programs to consider effective methods for teaching documentation skills and our workshop offers one approach that demonstrates effectiveness in enhancing knowledge, skills, and attitudes/behaviors.
The strengths of this study include: its foundation on the conceptual framework of deliberate practice with a higher-order educational outcome (ie, demonstration of learned skill)
; the development of a broadly applicable assessment checklist to assess documentation quality; innovation in its multifaceted educational approach for teaching and practicing documentation skills; methodologic grounding in the 6-step framework for curriculum design; and its multicenter pilot. The partnership between PCCM education experts and legal/risk management experts allowed for the module to be informed by multiple stakeholders with different perspectives on the issue. Finally, the creation of a full toolkit that is readily disseminated allows other medical educators a mechanism to address a common deficit in training programs.
Our study had design limitations. We did not include a long-term follow-up component to determine retention and endurance of acquired knowledge and skills. All participants were from the same pediatric subspecialty and findings may not be fully generalizable to other specialties and subspecialties. We attempted to mitigate this selection bias by creating a checklist for essential documentation elements that was informed by the broader literature and would span other fields. Further, this checklist was assessed for content and process validity as well as internal consistency. Additionally, we chose to analyze the first and fourth note because they represent baseline skill and skill at the end of the module. This decision is susceptible to confounders because we did not alter the order in which the types of vignettes are presented and therefore some changes observed might have been a result of the different clinical situations presented rather than trainee skill. This could be mitigated by altering the order in which the vignettes and videos are presented.
Future work should evaluate the efficacy of this approach in other learner groups and determine impact on long-term incorporation of documentation skills, while also determining if such training enhances medical documentation for actual patient care, with a possible impact on documentation time burden and provider well-being.
A multimodal interactive medical documentation curriculum had an impact in improving knowledge, skills, and attitudes/behaviors in PCCM fellows that may extend to other clinicians. This curriculum is now available to other medical educators.
Additional Site PIs:
Adrian Zurca, MD Penn State Health
Michael Green, MD University of Texas Southwestern
Toni Petrillo, MD Children's Hospital of Atlanta
Margaret Winkler, MD University of Alabama
Sanjiv Pasala, MD University of Arkansas for Medical Sciences
Tensing Maa, MD Nationwide Children's Hospital
We acknowledge the other members of the Education in Pediatric Intensive Care research collaborative: Grace Arteaga (Mayo Clinic, Scottsdale, Ariz); Meredith Bone (Northwestern University Feinberg School of Medicine, Chicago,Ill); Nana Coleman (Baylor College of Medicine, Houston, Tex); Stephanie Conrad (Vanderbilt University, Nashville,Tenn); Alvaro Coronado Munoz (Memorial Hermann Hospital, Houston, Tex); Angela Czaja (Department of Pediatrics, Division of Critical Care Medicine, University of Colorado School of Medicine, Aurora, Colo); Dennis Daniel (Boston Children's Hospital, Boston, Mass); Kim Derespina (Albert Einstein College of Medicine, Bronx, NY); Amanda Emke (Washington University School of Medicine, St. Louis, Mo); Melissa Evans (Medical University of South Carolina, Charleston, NC); Ryan Good (University of Colorado); Michael Green (University of Texas Southwestern Medical Center); Melinda Hamilton (Children's Hospital of Pittsburgh at UPMC, Pittsburgh, Pa); Natalie Henderson (University of Louisville, Louisville, Ky); Aaron Kessel (Cohen Children's Medical Center of NY, New Hyde Park, NY); Kevin Kuo (Stanford University, Stanford, Calif); Tensing Maa (Nationwide Children's Hospital, Columbus, Ohio); Karen Marcdante (Medical College of Wisconsin, Milwaukee, Wis); Vinai Modem (Memorial Hermann Hospital, Houston, Tex); Sanjiv Pasala (University of Arkansas School for Medical Sciences, Little Rock, Ariz); Toni Petrillo (Emory/Children's Hospital of Atlanta); Kyle Rehder (Duke University Hospital and Health System, Durham, NC); William Sasser (University of Alabama– Birmingham, Birmingham, Ala); Ashley Siems (Children's National Medical Center, Washington, DC); Sushant Srinivasan (University of Wisconsin, Madison, Wis); Marie Steiner (University of Minnesota, Minneapolis, Minn); Felice Su (Stanford University School of Medicine, Stanford, Calif); Markita Suttle (Nationwide Children's Hospital, Columbus, Ohio); Kelly Tieves (Children's Mercy Hospital, Kansas City, Mo); Shivani Tripathi (Memorial Hermann Hospital, Houston, Tex); and Theodore Wu (UT Health San Antonio, San Antonio, Tex).