If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Three years ago, the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Pediatrics spearheaded the Pediatrics Milestones Project as the next step toward achieving the promise of the Outcome Project.
A Working Group assembled to begin the process of exploring the literature to delineate the trajectory of competency development for each of 51 competencies comprising the 6 ACGME domains and a seventh, Personal and Professional Development, added by the pediatric education community. The group recognized that a successful outcome required application of continuous improvement methodology, using an iterative process to achieve the project's 3 main goals: 1) defining milestones for the ACGME competencies in the context of pediatrics; 2) establishing performance standards for the milestones; and 3) identifying assessment tools to be used across programs to allow benchmarking at a national level.
Reflecting on our charge, progress to date has resulted in near achievement of goal 1. Further progress will require engagement of educators and learners, using our current draft of the Pediatrics Milestones as the foundation. Therefore, our purposes here are: 1) to describe the process in the context of improvement science leading to the first complete iteration of the Pediatrics Milestones
to inform other specialties currently on the journey; 2) to serve as a springboard for discussion and to allow analysis, and testing that are prerequisites for effective implementation; 3) to delineate what the current product is and what it can be used for; 4) to caution about what the product is not and what it should not be mistaken for; and perhaps most importantly, 5) to get this work into the community so that others may both benefit from and improve it, and help us create the path forward.
Creating the Milestones: The Learning Cycles
The initial steps in our process have been outlined previously.
Table 1 outlines the full series of steps required to arrive at the release of the milestones draft. These steps align well with the precepts of improvement science as outlined in “The Improvement Guide.”
The initial steps in improvement science require learning about a proposed change using a variety of lenses, conversations, and hypotheses. Our initial learning cycles produced a set of Guiding Principles, several of which informed future steps. For example, to test the guiding principle that “the ACGME competencies are a necessary but (may) not (be) a sufficient framework for our work,” we engaged the Association of Pediatric Program Directors (APPD). On the basis of several years of effort to integrate the ACGME competencies into their training programs, we asked the APPD membership to reflect on what essential competencies for the practicing pediatrician were missing or were not explicit enough. The outcome was the identification of additional competencies in the domain of Personal and Professional Development.
With the Guiding Principles as our navigator, we reviewed the health, education, and social science literature to help us understand how individuals develop competencies that are required of practicing pediatricians. We then created draft backgrounds to serve as brief synopses of the literature we used to inform our understanding of each competency's developmental progression. On the basis of these backgrounds, we described a progression of behaviors that represent advancing levels of performance for that competency. Each successive level, then, represents a milestone, and the whole progression represents a series of milestones for each competency.
When using improvement methodology, the study segment of the “Plan–Do–Study–Act” cycle frequently reveals previously unanticipated outcomes. For example, we found substantial overlap in the literature for 2 of the competencies, resulting in a decision to combine them into a single set of milestones: the Medical Knowledge competency “critically evaluate and apply current medical information and scientific evidence for patient care” and the Practice-Based Learning and Improvement competency “locate, appraise, and assimilate evidence from scientific studies related to their patients' health problems.” In our final learning cycles, we sought input from content experts whose ideas had informed our understanding for many of the milestones.
What the Pediatrics Milestones are in Their Current State…
For each competency, we used the best evidence available to define the developmental progression from early learner to master physician. Each series of milestones, then, is our best educated hypothesis about how pediatricians progress from novice to master in a specific competency. The current document can therefore be used to inform one's thinking about competence development, and perhaps even frame both faculty and learners' thoughts about their own growth and development.
…and What the Pediatrics Milestones are Not
The Pediatrics Milestones are not yet actual milestones. Each of our series of milestones provides suggested behavioral anchors that indicate a level of performance for a given competency. One of the most important next steps will be testing them in vivo, to determine when in the educational trajectory most learners demonstrate behaviors consistent with a given milestone in the series. This process will resemble the approach to studying the Denver Developmental Screening Test,
and will allow us to learn whether observed behaviors confirm our proposed milestones and their sequence of progression, and when in the educational trajectory they should be expected to be achieved. Figure 1 illustrates how the milestones associated with a specific competency might fit the analogy to the Denver Developmental Scale. Defining the expected range of time and experience for reaching each of the milestones will also allow for early identification of learners in difficulty. Additionally, the nature of the behavioral descriptions should facilitate identification of a deficiency and assist in directing the remediation process. The milestones also can serve as a roadmap to mastery for learners who are at or above the expected level. Finally, although potentially valuable in formative feedback, each series of milestones is in effect a hypothesis and thus not ready to be used for high stakes assessment before: 1) establishment of evidence-based performance standards, 2) development of valid and reliable tools for their assessment, and 3) faculty development. Similarly, although we are confident that our process to date has yielded a great beginning, we are certain the current iteration is not what the final product will be. We may find, for example, despite our best educated hypotheses, that we have incorrectly characterized the developmental progression for the series of milestones for any given competency.
The Path Ahead
We hope that publishing our work to date and giving voice to the challenges that we face will inspire the reader to think differently about how physicians learn and develop, and to join us in the next phase of understanding how milestones contribute to advancing competency-based learning and assessment. Specifically, we will need to develop an assessment approach that takes into consideration the key attributes of an assessment tool outlined by Van der Vleuten's utility model.
This model suggests that an assessment tool's utility is equal to the product of 5 attributes: reliability, validity, acceptability, cost, and educational impact. This approach to assessment is going to require engagement of the medical education community to help us address each of these 5 attributes as they apply to milestones in toto and to the component parts. We will also need to understand and test the use of the Pediatrics Milestones in various contexts and settings, to address such questions as: to what extent are they generalizable to a variety of practice settings within pediatrics or to other specialties? Finally, we recognize that the faculty development needs around the milestones will be significant, as will the investment in time and continuity with learners required for meaningful assessment. Only thorough testing to establish the 5 attributes of an assessment tool, understanding their use in various settings, and developing faculty in their use, can the Pediatrics Milestones reach their potential to significantly move forward our approach to true competency-based education.
Before readers become overwhelmed by the work ahead and the potential burden on an evaluator, we ask that they consider the likely possibility that some milestones are not independent of each other, and thus potentially amenable to more streamlined assessment. In other words, how a learner is assessed on one series of milestones is likely to correlate with a number of others. As a result, we hope in the future to be able to reduce the number of competencies and their respective milestones from the current 51 to a much more manageable number, but this will only happen through in vivo study. For example, we previously reported that for several competencies and their series of milestones physician development progresses from dependence on external prompts, consequences or influences, such as oversight and regulation, to a more intrinsically driven, self-directed and self-regulated process, involving learner reflection for the purpose of improvement.
We can use this lesson as an example of how the milestones might result in more feasible strategies for assessment. One hypothesis is that performance in all of the competencies that follow this progression is measured by where one lies on the external–internal driver spectrum. Then assessment that a learner has reached a specific milestone on one of these competencies may be a proxy for all the others, serving as the critical link or common denominator. An assessment tool that has proven reliable and valid for one of these competencies may then potentially be utilized as predictive for all of the others. Some of the critical work ahead, then, is to explore these “linkage” hypotheses in the field.
Conclusion: A Call to Action
We hope that sharing the current draft of the Pediatrics Milestones at this juncture inspires educators and learners alike. On the individual level, enhancing understanding of the development of competence can only help to better identify root causes of learner difficulties and possible paths to improvement. Realizing the goal of advancing competency-based education through the Pediatrics Milestones Project will require collaboration of the medical education community to further develop and test the milestones and to learn from our successes and inevitable failures.
The good news is we are off to a great start! The APPD, through its Longitudinal Assessment Education Research Network (APPD/LEARN) and in partnership with the National Board of Medical Examiners is engaged in a pilot study of 9 of the milestones series. The hope is to have this pilot serve as a model for further study of milestones. To that end APPD/LEARN has also partnered with the Initiative for Innovation in Pediatric Education (IIPE) to create a new pathway to innovation that will invite programs to participate in centrally designed studies. The inaugural project will be a scale-up of the current pilot study of the milestones to include the 21 that will be reported to the ACGME as part of the Next Accreditation System. These 21 milestones were selected by the Pediatric Residency Review Committee with input from the APPD and Milestones Working Group.
All of this work, and the future projects we hope the readership will consider to test the milestones, underscore the fact that assessment in medical education is not a technical challenge with a straightforward solution, such as to create a series of milestones and then fast-forward into using them as a tool in high-stakes assessment. Rather, assessment in medical education is an adaptive challenge in which solutions require major shifts in our way of thinking and habits of practice.
Fast-forwarding is not a solution; to the contrary, in the spirit of continuous improvement, we will need to rewind and replay at each step in an iterative fashion, with each solution looking at both intended and unintended consequences as we try to move forward. We have a long way to go to achieve the broad scale implementation of these milestones in a manner that produces the meaningful and comprehensive approach to learning and assessment that we are all anxious to realize. We hope we have provided a solid foundation upon which to begin this dialogue.