Advertisement

The Pediatrics Milestones: A Continuous Quality Improvement Project is Launched—Now the Hard Work Begins!

      Three years ago, the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Pediatrics spearheaded the Pediatrics Milestones Project as the next step toward achieving the promise of the Outcome Project.

      Accreditation Council for Graduate Medical Education. ACGME general competency requirements, effective July 1, 2002. Available at: http://www.acgme.org/acWebsite/irc/irc_compintro.asp. Accessed January 17, 2012.

      A Working Group assembled to begin the process of exploring the literature to delineate the trajectory of competency development for each of 51 competencies comprising the 6 ACGME domains and a seventh, Personal and Professional Development, added by the pediatric education community. The group recognized that a successful outcome required application of continuous improvement methodology, using an iterative process to achieve the project's 3 main goals: 1) defining milestones for the ACGME competencies in the context of pediatrics; 2) establishing performance standards for the milestones; and 3) identifying assessment tools to be used across programs to allow benchmarking at a national level.
      Reflecting on our charge, progress to date has resulted in near achievement of goal 1. Further progress will require engagement of educators and learners, using our current draft of the Pediatrics Milestones as the foundation. Therefore, our purposes here are: 1) to describe the process in the context of improvement science leading to the first complete iteration of the Pediatrics Milestones

      Pediatrics Milestone Project. Available at: https://www.abp.org/abpwebsite/publicat/milestones.pdf. Accessed January 17, 2012.

      to inform other specialties currently on the journey; 2) to serve as a springboard for discussion and to allow analysis, and testing that are prerequisites for effective implementation; 3) to delineate what the current product is and what it can be used for; 4) to caution about what the product is not and what it should not be mistaken for; and perhaps most importantly, 5) to get this work into the community so that others may both benefit from and improve it, and help us create the path forward.

      Creating the Milestones: The Learning Cycles

      The initial steps in our process have been outlined previously.
      • Englander R.
      • Hicks P.
      • Benson B.
      • et al.
      Pediatrics Milestones: a developmental approach to the competencies.
      • Hicks P.J.
      • Schumacher D.J.
      • Benson B.J.
      • et al.
      The Pediatric Milestones: conceptual framework, guiding principles, and approach to development.
      • Hicks P.J.
      • Englander R.
      • Schumacher D.J.
      • et al.
      Pediatric milestone project: next steps toward meaningful outcomes assessment.
      Table 1 outlines the full series of steps required to arrive at the release of the milestones draft. These steps align well with the precepts of improvement science as outlined in “The Improvement Guide.”
      • Langley G.
      • Moen R.
      • Nolan K.
      • et al.
      The Improvement Guide: A Practical Approach to Enhancing Organizational Performance.
      We will highlight a couple of these steps as examples of how we used an iterative improvement model to get to the current product.
      Table 1Steps in the Process of Developing the Pediatric Milestones
      Foundational Work (Before Milestones Creation)
      1Assemble a Working Group that includes medical educators and at least one trainee, with content expertise in the ACGME competencies and experience as medical educators.
      2Assemble an Advisory Board representing members of the educational continuum, assessment experts, and thought leaders within the specialty community.
      3Develop Guiding Principles and a Conceptual Framework.
      4Engage specialty-specific program directors’ association to re-examine ACGME competency framework for gaps.
      Steps in Creating the Milestones First Draft
      5Designate a primary author for the creation of milestones for each subcompetency.
      6Primary authors perform a review of the literature from health and social sciences to determine developmental progression of Knowledge, Skills, and Attitudes for each subcompetency.
      7Primary authors develop a synopsis of the literature and draft of milestones for each subcompetency.
      8Primary authors vet their work with the Working Group (this required every-other-week conference calls over an 18 month period).
      9Primary authors edit based on Working Group feedback.
      10Edited milestones are revisited by the Working Group (this required every-other-week conference calls over a 6-month period).
      Steps in Creating the Milestones Second Draft: External Review
      11Content experts are asked to review the milestones for selected subcompetencies.
      12Primary authors make edits based on content expert feedback.
      13The Working Group reviews the edited milestone documents (this required a series of weekly calls over 2 months).
      Steps in Creating the Final Draft: Editing and Publication
      14The completed draft is submitted to a copy editor for review.
      15The Chair of the project edits the final document with attention to both copy and content.
      16The Pediatric Milestones are published on the American Board of Pediatrics Web site.
      ACGME = Accreditation Council for Graduate Medical Education.
      The initial steps in improvement science require learning about a proposed change using a variety of lenses, conversations, and hypotheses. Our initial learning cycles produced a set of Guiding Principles, several of which informed future steps. For example, to test the guiding principle that “the ACGME competencies are a necessary but (may) not (be) a sufficient framework for our work,” we engaged the Association of Pediatric Program Directors (APPD). On the basis of several years of effort to integrate the ACGME competencies into their training programs, we asked the APPD membership to reflect on what essential competencies for the practicing pediatrician were missing or were not explicit enough. The outcome was the identification of additional competencies in the domain of Personal and Professional Development.

      Pediatrics Milestone Project. Available at: https://www.abp.org/abpwebsite/publicat/milestones.pdf. Accessed January 17, 2012.

      With the Guiding Principles as our navigator, we reviewed the health, education, and social science literature to help us understand how individuals develop competencies that are required of practicing pediatricians. We then created draft backgrounds to serve as brief synopses of the literature we used to inform our understanding of each competency's developmental progression. On the basis of these backgrounds, we described a progression of behaviors that represent advancing levels of performance for that competency. Each successive level, then, represents a milestone, and the whole progression represents a series of milestones for each competency.
      When using improvement methodology, the study segment of the “Plan–Do–Study–Act” cycle frequently reveals previously unanticipated outcomes. For example, we found substantial overlap in the literature for 2 of the competencies, resulting in a decision to combine them into a single set of milestones: the Medical Knowledge competency “critically evaluate and apply current medical information and scientific evidence for patient care” and the Practice-Based Learning and Improvement competency “locate, appraise, and assimilate evidence from scientific studies related to their patients' health problems.” In our final learning cycles, we sought input from content experts whose ideas had informed our understanding for many of the milestones.

      What the Pediatrics Milestones are in Their Current State…

      The link in this article makes available the most recent iteration of the Pediatrics Milestones (https://www.abp.org/abpwebsite/publicat/milestones.pdf).

      Pediatrics Milestone Project. Available at: https://www.abp.org/abpwebsite/publicat/milestones.pdf. Accessed January 17, 2012.

      For each competency, we used the best evidence available to define the developmental progression from early learner to master physician. Each series of milestones, then, is our best educated hypothesis about how pediatricians progress from novice to master in a specific competency. The current document can therefore be used to inform one's thinking about competence development, and perhaps even frame both faculty and learners' thoughts about their own growth and development.

      …and What the Pediatrics Milestones are Not

      The Pediatrics Milestones are not yet actual milestones. Each of our series of milestones provides suggested behavioral anchors that indicate a level of performance for a given competency. One of the most important next steps will be testing them in vivo, to determine when in the educational trajectory most learners demonstrate behaviors consistent with a given milestone in the series. This process will resemble the approach to studying the Denver Developmental Screening Test,
      • Frankenburg W.K.
      • Dodds J.B.
      The Denver Developmental Screening Test.
      and will allow us to learn whether observed behaviors confirm our proposed milestones and their sequence of progression, and when in the educational trajectory they should be expected to be achieved. Figure 1 illustrates how the milestones associated with a specific competency might fit the analogy to the Denver Developmental Scale. Defining the expected range of time and experience for reaching each of the milestones will also allow for early identification of learners in difficulty. Additionally, the nature of the behavioral descriptions should facilitate identification of a deficiency and assist in directing the remediation process. The milestones also can serve as a roadmap to mastery for learners who are at or above the expected level. Finally, although potentially valuable in formative feedback, each series of milestones is in effect a hypothesis and thus not ready to be used for high stakes assessment before: 1) establishment of evidence-based performance standards, 2) development of valid and reliable tools for their assessment, and 3) faculty development. Similarly, although we are confident that our process to date has yielded a great beginning, we are certain the current iteration is not what the final product will be. We may find, for example, despite our best educated hypotheses, that we have incorrectly characterized the developmental progression for the series of milestones for any given competency.
      Figure thumbnail gr1
      Figure 1Example of time course of achievement of competency developmental milestones.

      The Path Ahead

      We hope that publishing our work to date and giving voice to the challenges that we face will inspire the reader to think differently about how physicians learn and develop, and to join us in the next phase of understanding how milestones contribute to advancing competency-based learning and assessment. Specifically, we will need to develop an assessment approach that takes into consideration the key attributes of an assessment tool outlined by Van der Vleuten's utility model.
      • van der Vleuten C.P.M.
      • Schuwirth L.W.T.
      Assessing professional competence: from methods to programmes.
      This model suggests that an assessment tool's utility is equal to the product of 5 attributes: reliability, validity, acceptability, cost, and educational impact. This approach to assessment is going to require engagement of the medical education community to help us address each of these 5 attributes as they apply to milestones in toto and to the component parts. We will also need to understand and test the use of the Pediatrics Milestones in various contexts and settings, to address such questions as: to what extent are they generalizable to a variety of practice settings within pediatrics or to other specialties? Finally, we recognize that the faculty development needs around the milestones will be significant, as will the investment in time and continuity with learners required for meaningful assessment. Only thorough testing to establish the 5 attributes of an assessment tool, understanding their use in various settings, and developing faculty in their use, can the Pediatrics Milestones reach their potential to significantly move forward our approach to true competency-based education.
      Before readers become overwhelmed by the work ahead and the potential burden on an evaluator, we ask that they consider the likely possibility that some milestones are not independent of each other, and thus potentially amenable to more streamlined assessment. In other words, how a learner is assessed on one series of milestones is likely to correlate with a number of others. As a result, we hope in the future to be able to reduce the number of competencies and their respective milestones from the current 51 to a much more manageable number, but this will only happen through in vivo study. For example, we previously reported that for several competencies and their series of milestones physician development progresses from dependence on external prompts, consequences or influences, such as oversight and regulation, to a more intrinsically driven, self-directed and self-regulated process, involving learner reflection for the purpose of improvement.
      • Englander R.
      • Hicks P.
      • Benson B.
      • et al.
      Pediatrics Milestones: a developmental approach to the competencies.
      We can use this lesson as an example of how the milestones might result in more feasible strategies for assessment. One hypothesis is that performance in all of the competencies that follow this progression is measured by where one lies on the external–internal driver spectrum. Then assessment that a learner has reached a specific milestone on one of these competencies may be a proxy for all the others, serving as the critical link or common denominator. An assessment tool that has proven reliable and valid for one of these competencies may then potentially be utilized as predictive for all of the others. Some of the critical work ahead, then, is to explore these “linkage” hypotheses in the field.

      Conclusion: A Call to Action

      We hope that sharing the current draft of the Pediatrics Milestones at this juncture inspires educators and learners alike. On the individual level, enhancing understanding of the development of competence can only help to better identify root causes of learner difficulties and possible paths to improvement. Realizing the goal of advancing competency-based education through the Pediatrics Milestones Project will require collaboration of the medical education community to further develop and test the milestones and to learn from our successes and inevitable failures.
      The good news is we are off to a great start! The APPD, through its Longitudinal Assessment Education Research Network (APPD/LEARN) and in partnership with the National Board of Medical Examiners is engaged in a pilot study of 9 of the milestones series. The hope is to have this pilot serve as a model for further study of milestones. To that end APPD/LEARN has also partnered with the Initiative for Innovation in Pediatric Education (IIPE) to create a new pathway to innovation that will invite programs to participate in centrally designed studies. The inaugural project will be a scale-up of the current pilot study of the milestones to include the 21 that will be reported to the ACGME as part of the Next Accreditation System. These 21 milestones were selected by the Pediatric Residency Review Committee with input from the APPD and Milestones Working Group.
      All of this work, and the future projects we hope the readership will consider to test the milestones, underscore the fact that assessment in medical education is not a technical challenge with a straightforward solution, such as to create a series of milestones and then fast-forward into using them as a tool in high-stakes assessment. Rather, assessment in medical education is an adaptive challenge in which solutions require major shifts in our way of thinking and habits of practice.
      • Heifetz R.A.
      • Laurie D.L.
      The work of leadership.
      Fast-forwarding is not a solution; to the contrary, in the spirit of continuous improvement, we will need to rewind and replay at each step in an iterative fashion, with each solution looking at both intended and unintended consequences as we try to move forward. We have a long way to go to achieve the broad scale implementation of these milestones in a manner that produces the meaningful and comprehensive approach to learning and assessment that we are all anxious to realize. We hope we have provided a solid foundation upon which to begin this dialogue.

      References

      1. Accreditation Council for Graduate Medical Education. ACGME general competency requirements, effective July 1, 2002. Available at: http://www.acgme.org/acWebsite/irc/irc_compintro.asp. Accessed January 17, 2012.

      2. Pediatrics Milestone Project. Available at: https://www.abp.org/abpwebsite/publicat/milestones.pdf. Accessed January 17, 2012.

        • Englander R.
        • Hicks P.
        • Benson B.
        • et al.
        Pediatrics Milestones: a developmental approach to the competencies.
        J Pediatr. 2010; 157: 521-522
        • Hicks P.J.
        • Schumacher D.J.
        • Benson B.J.
        • et al.
        The Pediatric Milestones: conceptual framework, guiding principles, and approach to development.
        J Grad Med Educ. 2010; 2: 410-418
        • Hicks P.J.
        • Englander R.
        • Schumacher D.J.
        • et al.
        Pediatric milestone project: next steps toward meaningful outcomes assessment.
        J Grad Med Educ. 2010; 2: 577-584
        • Langley G.
        • Moen R.
        • Nolan K.
        • et al.
        The Improvement Guide: A Practical Approach to Enhancing Organizational Performance.
        2nd ed. Jossey-Bass, San Francisco, Calif2009
        • Frankenburg W.K.
        • Dodds J.B.
        The Denver Developmental Screening Test.
        J Pediatr. 1967; 71: 181
        • van der Vleuten C.P.M.
        • Schuwirth L.W.T.
        Assessing professional competence: from methods to programmes.
        Med Educ. 2005; 39: 309-317
        • Heifetz R.A.
        • Laurie D.L.
        The work of leadership.
        Harvard Bus Rev, 2001 (Dec) (1–13)