Advertisement

Integrating the Learner's Perspective in the Refinement of Competency-Based Assessments

      Beginning with the Accreditation Council for Graduate Medical Education's (ACGME) Outcomes Project, which introduced the core competency domains within medical practice,
      • Swing S.
      The ACGME outcome project: retrospective and prospective.
      there has been a shift in graduate medical education towards competency-based training.
      • Carraccio C.
      • Wolfsthal S.D.
      • Englander R.
      • et al.
      Shifting paradigms: from Flexner to competencies.
      Ideally, this framework aligns the medical needs of patients with ensuring that the outcomes of training prepare graduates to meet those needs
      • Frenk J.
      • Chen L.
      • Bhutta Z.
      • et al.
      Health professionals for a new century: transforming education to strengthen health systems in an interdependent world.
      —a goal consistent with ongoing efforts to improve the quality and safety of patient care.
      Committee on Quality of Health Care in America; Institute of Medicine
      Crossing the Quality Chasm: A New Health System for the 21st Century.
      National Research Council
      Health Professions Education: A Bridge to Quality.
      Educators have responded to concerns about the frequency of medical errors and the quality of medical training by focusing assessment strategies on the transparent, accurate documentation of trainee performance within key clinical competency domains.
      • Swing S.R.
      • Clyman S.G.
      • Holmboe E.S.
      • et al.
      Advancing resident assessment in graduate medical education.
      Currently, competency-based assessment revolves around the tracking of progression on developmental milestones
      • Hicks P.J.
      • Schumacher D.J.
      • Benson B.J.
      • et al.
      The Pediatrics Milestones: conceptual framework, guiding principles, and approach to development.
      —following learner development across a continuum, from novice to independently trustworthy practitioner to expert. These assessments can also provide trainees with key data that can be used in identifying areas of proficiency and weakness, a critically important aspect in the process of informed self-assessment.
      • Sargeant J.
      • Armson H.
      • Chesluk B.
      • et al.
      The process and dimensions of informed self-assessment: a conceptual model.
      As Eva and Regehr
      • Eva K.
      • Regehr G.
      Self-assessment in the health professions: a reformulation and research agenda.
      state, “The route to self-improvement is not through becoming a more accurate self-assessor, but through seeking out feedback from reliable and valid external sources . . . making a special effort to take the resulting feedback seriously rather than discounting it: to reflect rather than ruminate.”
      Postgraduate pediatric learners are key stakeholders and consumers of the formal feedback and evaluative data that competency-based assessments provide. Although there are many perspectives from expert clinicians and educators for how to create valid, reliable competency-based assessments of pediatric trainees,
      • Carraccio C.
      • Englander R.
      • Wolfsthal S.
      • et al.
      Educating the pediatrician of the 21st century: defining and implementing a competency-based system.
      • Hicks P.J.
      • Englander R.
      • Schumacher D.J.
      • et al.
      Pediatrics Milestone Project: next steps toward meaningful outcomes assessment.
      the learner's viewpoint has largely been unsolicited.
      • Watling C.
      • Lingard L.
      Toward meaningful evaluation of medical trainees: the influence of participants' perceptions of the process.
      To be most meaningful, assessment and feedback systems must be structured in a manner that aids the learner's engagement in self-assessment—evincing a roadmap of professional development and guiding their self-improvement efforts.
      • Hicks P.J.
      • Schumacher D.J.
      • Benson B.J.
      • et al.
      The Pediatrics Milestones: conceptual framework, guiding principles, and approach to development.
      • Schumacher D.
      • Englander R.
      • Carraccio C.
      Developing the master learner: applying learning theory to the learner, the teacher, and the learning environment.

      From the Horse's Mouth: Resident Perceptions of Formal Feedback

      As we continue to develop competency-based assessment tools, it is important to consider how postgraduate learners in pediatrics perceive and integrate formal assessment data. Although face-to-face, in-person feedback from clinical supervisors continues to be a key component of formative feedback for postgraduate trainees, the content of these sessions is difficult to document. Rather, we sought to study and better understand learner's perception and utilization of formal written assessments. We collected opinions from trainees at a single, large pediatric residency program. Our goals were to better understand what aspects of written performance assessments are most (and least) useful to trainees in their professional development; and to consider how competency-based assessments can be adapted to provide data that informs learners' self-appraisal and direction of subsequent actions to address areas in need of improvement.

      Data Source

      We surveyed the 120 resident physicians enrolled in the Boston Combined Residency Program (BCRP) during the academic year 2011–2012. The institutional review board at Boston Children's Hospital reviewed this project and deemed it exempt.
      At the time of this survey, the BCRP used global rating scale assessments, eliciting faculty ratings of trainee performance upon completion of a clinical rotation. Ratings on each of the 6 domains of competence were developed and documented using a 9-point Likert scale. Behavioral anchors were provided at the extremes of the scale. Mandatory free-text sections for overall assessment of performance on the rotation were also included in each assessment. Other assessment strategies employed during this period included simulation, record review, and 360-degree evaluations.

      Focus Groups

      We initially conducted two 60-minute focus groups of 10 to 12 residents each, with equal representation from each postgraduate year (PGY) level. A medical education research expert not affiliated with the training program facilitated these sessions. Residents were asked to discuss their perceptions of data received via formal assessments. Subsequently, they were asked to consider how feedback modalities could be adapted to aid their identification of clinical strengths and weaknesses and guide efforts for self-improvement. The facilitator reviewed prompts and probes with the study authors before the sessions and had autonomy to explore themes that were identified during the sessions.
      Several themes emerged. First, residents echoed previously published findings stating that numeric ratings on broad domains of competence were difficult to interpret.
      • Lurie S.J.
      • Mooney C.J.
      • Lyness J.M.
      Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: a systematic review.
      They expressed concern that numeric ratings on general competencies lacked the work-based, clinical context required to help them determine whether their performance was appropriate for their level of training. Also, these ratings did not assist them in identifying areas of weakness; residents struggled to formulate a plan to address low scores across broad competency domains such as professionalism. Some residents voiced concern that they were assessed on competencies that were infrequently observed by supervisors, such as systems-based practice, or were from evaluators with whom they spent little time. Last, residents nearing the end of their training desired more definitive formal assessment on whether their supervisors trusted them and whether they were nearing the level of being able to practice without supervision.

      Survey

      Drawing on themes from the focus group discussions, we distributed an electronic survey to all pediatric residents in the program to elicit viewpoints on assessment modalities, focusing on how assessments could be adapted to provide more useful feedback. Survey items were either dichotomous (yes/no) or were ranked on a 5-point Likert scale. Tests of significance were conducted using chi-square, linear regression or analysis of variance, as appropriate. Survey data analysis was completed using Stata SE 13.1 software (StataCorp, College Station, Tex).
      The overall survey response rate was 68% (n = 82), with the highest response rate coming from PGY-3s (PGY-1: 58%, PGY-2: 68%, PGY-3: 80%). Perceived utility of rating scale assessments decreased significantly as residents progressed in training (P = .02). Residents found assessments were most useful when coming from the attending with whom they spent the most time on service, or from a composite of attendings they worked with on shift-based rotations (eg, emergency department). Residents rated the utility of other forms of rotation-specific assessment, such as simulation or review of written records, significantly higher than numeric data from end of rotation ratings (P < .01).
      Although residents were dissatisfied with ratings on the general competencies, 94% of residents stated that they would prefer ratings if they were based on patient care activities specific to the clinical rotation being assessed. When asked which components of assessments were most useful to their professional development, residents rated importance of free text comments highly, followed closely by numeric ratings on rotation specific skills (eg, procedures within the emergency department or running a family meeting on oncology) (Figure).
      Figure thumbnail gr1
      FigureResident ratings of evaluation components. *Importance rating differs significantly across evaluation components (ANOVA, P < .05).

      Applying Lessons Learned to Competency-Based Assessments: Closing the Gap

      Great strides have been made in the development of competency-based assessments of pediatric trainees that objectively document trainees' progression towards readiness for safe, high-quality practice.
      • Englander R.
      • Burke A.
      • Guralnick S.
      • et al.
      The Pediatrics Milestones: a continuous quality improvement project is launched—now the hard work begins!.
      However, learner perspectives must be integrated into assessment building efforts so that data can be effectively synthesized and applied in the learner's developmental process. The gap between assessment of and assessment for trainees can be closed considerably by utilizing several key perspectives from the learner, including the integration of work-based context into assessments and ensuring that feedback is elicited from a trusted, recognized source.

      Embracing Context in Reconsidering Assessments

      On the basis of our findings, we believe that ratings on general competencies should be deemphasized and replaced with clinically contextualized feedback. Grounding assessments in the context of real work output allows feedback to be more tangible, meeting faculty and residents where they practice: at the bedside. Consider pediatric intensivists assessing trainees on their management of a patient with respiratory failure. In addition to considering the individual competencies and milestones involved in this practice-specific behavior, assessors also see the greater picture; they may draw conclusions regarding whether the trainee was able to put it all together and earn their trust in providing care for one of their patients.
      • ten Cate O.
      Trust, competence, and the supervisor's role in postgraduate training.
      This conclusion—that a trainee is trustworthy—is a key area of assessment,
      • Hauer K.
      • ten Cate O.
      • Boscardin C.
      • et al.
      Understanding trust as an essential element of trainee supervision and learning in the workplace.
      and one that we found to be very meaningful for trainees. Further, assessments focusing on specific clinical tasks, rather than broad domains of clinical performance, are more likely to have sustained influence in changing trainee behavior.
      • Brydges R.
      • Carnahan H.
      • Safir O.
      • et al.
      How effective is self-guided learning of clinical technical skills? It's all about process.
      The process of identifying, describing and evaluating these contextualized experiences is similar to work done in a single internal medicine residency which created rotation-specific, as well as rotation-independent, observable practice activities (OPAs).
      • Warm E.
      • Mathis B.
      • Held J.
      • et al.
      Entrustment and mapping of observable practice activities for resident assessment.
      Trainees in our study favored a transition of rating scale assessments from context-free to context based. They also placed high importance on feedback received via less commonly used assessment modalities, such as simulation and written record review. These adjunct assessment modalities can be effective in competency-based assessment
      • Swing S.
      Assessing the ACGME general competencies: general considerations and assessment methods.
      and often provide the targeted, clinically focused feedback trainees' desire. Further, prior research shows that assessment approaches that engage the assessor in their specific area of expertise may improve the reliability and validity of the data.
      • Turnbull J.
      • MacFadyen J.
      • Van Barneveld C.
      • et al.
      Clinical work sampling: a new approach to the problem of in-training evaluation.
      • Winckel C.P.
      • Reznick R.K.
      • Cohen R.
      • et al.
      Reliability and construct validity of a structured technical skills assessment form.
      These more specific ratings may also avoid several common pitfalls of rating scales, including halo effect, range restriction, and cognitive distortion.
      • Gray J.D.
      Global rating scales in residency education.
      Assessments that place a trainees' clinical performance in real work context should lead to richness well beyond the broad conclusions that are characteristic of global competency assessments.
      • Silber C.G.
      • Nasca T.J.
      • Paskin D.L.
      • et al.
      Do global rating forms enable program directors to assess the ACGME competencies?.
      These assessments will also better inform the creation of individualized learning plans, aiding both the learner and clinical mentor in identifying areas of deficiency and making a targeted plan for improvement.

      Ensure Feedback Comes from a Known, Invested Source

      While we found that trainees use feedback based on directly observed clinical activities to identify areas for self-improvement, they also expressed a desire to feel the assessor's engagement in the assessment process—a previously identified factor in helping trainees see feedback as credible
      • Bing-You R.
      • Trowbridge R.
      Why medical educators may be failing at feedback.
      and motivating trainees toward behavioral change.
      • Watling C.J.
      • Kenyon C.F.
      • Zibrowski E.M.
      • et al.
      Rules of engagement: residents' perceptions of the in-training evaluation process.
      With increasing demands on academic faculty leading to decreased time spent at the bedside in direct observation of trainee performance,
      • Ende J.
      Feedback in clinical medical education.
      learners may develop incorrect, often inflated, perceptions of their own abilities.
      • Davis D.A.
      • Mazmanian P.E.
      • Fordis M.
      • et al.
      Accuracy of physician self-assessment compared with observed measures of competence: a systematic review.
      This is magnified when negative assessment is given, as below-average assessments may be dismissed as inaccurate if coming from an evaluator who is not perceived as credible.
      • Higgins R.S.D.
      • Bridges J.
      • Burke J.M.
      • et al.
      Implementing the ACGME general competencies in a cardiothoracic surgery residency program using 360-degree feedback.
      Our findings suggest that feedback was most impactful when coming from a faculty member who had provided frequent, direct observation of the trainees' performance.
      Robust, clearly explained written comments, which cite specific findings from direct observations, should be a focus of all competency-based assessment modalities. It is clear that trainees struggle to find meaning in numeric ratings without the detailing of the supervisor's thought process. By asking faculty to discuss areas of particular strength and weakness in all assessments, negative feedback is normalized and perceived as constructive. Learners can expect to take away several areas for improvement from every clinical experience during training, thereby reinforcing the emphasis on lifelong learning and guided self-improvement. Educating assessors on the value of their narrative comments for both trainees and training program directors is a key target area for faculty development.

      Hope for the Future: Entrustable Professional Activities

      One promising approach to focusing assessment on observed performance, within a work-based context, is the use of entrustable professional activities (EPAs). EPAs value the clinical judgment of the expert evaluator, using direct observation of the trainee's delivery of care to make an assessment of where the trainee is in their progression toward unsupervised practice.
      • ten Cate O.
      Entrustability of professional activities and competency-based training.
      EPAs include many of the facets of assessments trainees described as useful, incorporating practice-specific feedback from invested assessors and giving the trainee a sense of their current level of entrustability to provide patient care. Although EPAs can be leveraged to provide trainees with the clinically based feedback they desire, they can also be judiciously mapped to milestones and competencies.
      • ten Cate O.
      • Scheele F.
      Competency-based postgraduate training: can we bridge the gap between theory and clinical practice?.
      By earning a statement of awarded responsibility on an EPA, trainees will have also demonstrated proficient performance, by proxy, on the key milestones and competencies therein. Within pediatrics and its associated subspecialities, EPAs are broadly defined, limited to the 10 to 15 activities that truly define the field. OPAs may be a way to operationalize these concepts in the creation of learner-centric assessments, though prudence will be required to avoid creating an exhaustive list of OPAs or linked sub-EPAs, which may be overly burdensome for programs to assess.
      • Teherani A.
      • Chen H.
      The next steps in competency-based medical education: milestones, entrustable professional activities and observable practice activities.
      Ultimately, 15 years after the introduction of the ACGME Outcomes Project, our goals remain the same: to produce competent physicians via a training system that is accountable to the general public it serves. Achieving this goal must have at its foundation a learner-centered approach that provides a road map for trainee development, with clear benchmarks to strive for and defined steps that lead up to key outcomes. Milestones and EPAs are well positioned to aid in achievement of this goal, but only if we use them in a way that has meaning to trainees and subsequently drives their learning.

      Acknowledgments

      This research was funded by a Fredrick Lovejoy Research Award . Sonja Ziniel, Children's Hospital Boston, aided in designing and conducting the focus groups.

      References

        • Swing S.
        The ACGME outcome project: retrospective and prospective.
        Med Teach. 2007; 29: 648-654
        • Carraccio C.
        • Wolfsthal S.D.
        • Englander R.
        • et al.
        Shifting paradigms: from Flexner to competencies.
        Acad Med. 2002; 77: 361-367
        • Frenk J.
        • Chen L.
        • Bhutta Z.
        • et al.
        Health professionals for a new century: transforming education to strengthen health systems in an interdependent world.
        Lancet. 2010; 376: 1923-1958
        • Committee on Quality of Health Care in America; Institute of Medicine
        Crossing the Quality Chasm: A New Health System for the 21st Century.
        National Academies Press, Washington, DC2001
        • National Research Council
        Health Professions Education: A Bridge to Quality.
        National Academies Press, Washington, DC2003
        • Swing S.R.
        • Clyman S.G.
        • Holmboe E.S.
        • et al.
        Advancing resident assessment in graduate medical education.
        J Grad Med Educ. 2009; 1: 278-286
        • Hicks P.J.
        • Schumacher D.J.
        • Benson B.J.
        • et al.
        The Pediatrics Milestones: conceptual framework, guiding principles, and approach to development.
        J Grad Med Educ. 2010; 2: 410-418
        • Sargeant J.
        • Armson H.
        • Chesluk B.
        • et al.
        The process and dimensions of informed self-assessment: a conceptual model.
        Acad Med. 2010; 85: 1212-1220
        • Eva K.
        • Regehr G.
        Self-assessment in the health professions: a reformulation and research agenda.
        Acad Med. 2005; 80: S46-S54
        • Carraccio C.
        • Englander R.
        • Wolfsthal S.
        • et al.
        Educating the pediatrician of the 21st century: defining and implementing a competency-based system.
        Pediatrics. 2004; 113: 252-258
        • Hicks P.J.
        • Englander R.
        • Schumacher D.J.
        • et al.
        Pediatrics Milestone Project: next steps toward meaningful outcomes assessment.
        J Grad Med Educ. 2010; 2: 577-584
        • Watling C.
        • Lingard L.
        Toward meaningful evaluation of medical trainees: the influence of participants' perceptions of the process.
        Adv Health Sci Educ. 2012; 17: 183-194
        • Schumacher D.
        • Englander R.
        • Carraccio C.
        Developing the master learner: applying learning theory to the learner, the teacher, and the learning environment.
        Acad Med. 2013; 88: 1635-1645
        • Lurie S.J.
        • Mooney C.J.
        • Lyness J.M.
        Measurement of the general competencies of the Accreditation Council for Graduate Medical Education: a systematic review.
        Acad Med. 2009; 84: 301-309
        • Englander R.
        • Burke A.
        • Guralnick S.
        • et al.
        The Pediatrics Milestones: a continuous quality improvement project is launched—now the hard work begins!.
        Acad Pediatr. 2012; 12: 471-474
        • ten Cate O.
        Trust, competence, and the supervisor's role in postgraduate training.
        BMJ. 2006; 333: 748-751
        • Hauer K.
        • ten Cate O.
        • Boscardin C.
        • et al.
        Understanding trust as an essential element of trainee supervision and learning in the workplace.
        Adv Health Sci Educ Theory Pract. 2014; 19: 435-456
        • Brydges R.
        • Carnahan H.
        • Safir O.
        • et al.
        How effective is self-guided learning of clinical technical skills? It's all about process.
        Med Educ. 2009; 43: 507-515
        • Warm E.
        • Mathis B.
        • Held J.
        • et al.
        Entrustment and mapping of observable practice activities for resident assessment.
        J Gen Intern Med. 2014; 29: 1177-1182
        • Swing S.
        Assessing the ACGME general competencies: general considerations and assessment methods.
        Acad Emerg Med. 2002; 9: 1278-1288
        • Turnbull J.
        • MacFadyen J.
        • Van Barneveld C.
        • et al.
        Clinical work sampling: a new approach to the problem of in-training evaluation.
        J Gen Intern Med. 2000; 15: 556-561
        • Winckel C.P.
        • Reznick R.K.
        • Cohen R.
        • et al.
        Reliability and construct validity of a structured technical skills assessment form.
        Am J Surg. 1994; 167: 423-427
        • Gray J.D.
        Global rating scales in residency education.
        Acad Med. 1996; 71: S55-S63
        • Silber C.G.
        • Nasca T.J.
        • Paskin D.L.
        • et al.
        Do global rating forms enable program directors to assess the ACGME competencies?.
        Acad Med. 2004; 79: 549-556
        • Bing-You R.
        • Trowbridge R.
        Why medical educators may be failing at feedback.
        JAMA. 2009; 302: 1330-1331
        • Watling C.J.
        • Kenyon C.F.
        • Zibrowski E.M.
        • et al.
        Rules of engagement: residents' perceptions of the in-training evaluation process.
        Acad Med. 2008; 83: S97-S100
        • Ende J.
        Feedback in clinical medical education.
        JAMA. 1983; 250: 777-781
        • Davis D.A.
        • Mazmanian P.E.
        • Fordis M.
        • et al.
        Accuracy of physician self-assessment compared with observed measures of competence: a systematic review.
        JAMA. 2006; 296: 1094-1102
        • Higgins R.S.D.
        • Bridges J.
        • Burke J.M.
        • et al.
        Implementing the ACGME general competencies in a cardiothoracic surgery residency program using 360-degree feedback.
        Ann Thorac Surg. 2004; 77: 12-17
        • ten Cate O.
        Entrustability of professional activities and competency-based training.
        Acad Med. 2005; 39: 1176-1177
        • ten Cate O.
        • Scheele F.
        Competency-based postgraduate training: can we bridge the gap between theory and clinical practice?.
        Acad Med. 2007; 82: 542-547
        • Teherani A.
        • Chen H.
        The next steps in competency-based medical education: milestones, entrustable professional activities and observable practice activities.
        J Gen Intern Med. 2014; 29: 1090-1092