Advertisement

The Pediatrics Milestones: Initial Evidence for Their Use as Learning Road Maps for Residents

Published:November 19, 2012DOI:https://doi.org/10.1016/j.acap.2012.09.003

      Abstract

      Objective

      As the next step in competency-based medical education, the Pediatrics Milestone Project seeks to provide a learner-centered approach to training and assessment. To help accomplish this goal, this study sought to determine how pediatric residents understand, interpret, and respond to the Pediatrics Milestones.

      Methods

      Cognitive interviews with 48 pediatric residents from all training levels at 2 training programs were conducted. Each participant reviewed one Pediatrics Milestone document (PMD). Eight total Pediatrics Milestones, chosen for their range of complexity, length, competency domain, and primary author, were included in this study. Six residents, 2 from each year of residency training, reviewed each PMD. Interviews were transcribed and coded using inductive methods, and codes were grouped into themes that emerged.

      Results

      Four major themes emerged through coding and analysis: 1) the participants' degree of understanding of the PMDs is sufficient, often deep; 2) the etiology of participants' understanding is rooted in their experiences; 3) there are qualities of the PMD that may contribute to or detract from understanding; and 4) participants apply their understanding by noting the PMD describes a developmental progression that can provide a road map for learning. Additionally, we learned that residents are generally comfortable being placed in the middle of a series of developmental milestones. Two minor themes focusing on interest and practicality were also identified.

      Conclusions

      This study provides initial evidence for the Pediatrics Milestones as learner-centered documents that can be used for orientation, education, formative feedback, and, ultimately, assessment.

      Keywords

      What's New
      As the next step in competency-based medical education, the Pediatrics Milestone Project seeks to provide a learner-centered approach to training and assessment. This study provides preliminary evidence for the Pediatrics Milestones as learner-centered documents useful for orientation, education, formative feedback, and ultimately assessment.
      A collaborative effort of the Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Pediatrics, the Pediatrics Milestone Project is the next step in advancing competency-based medical education (CBME).
      • Nasca T.J.
      • Philibert I.
      • Brigham T.
      • et al.
      The next GME accreditation system—rationale and benefits.
      • Hicks P.J.
      • Schumacher D.
      • Benson B.
      • et al.
      The Pediatrics Milestones: conceptual framework, guiding principles, and approach to development.
      • Hicks P.J.
      • Englander R.
      • Schumacher D.J.
      • et al.
      Pediatrics Milestone Project: Next Steps Toward Meaningful Outcomes.
      Its goals include: 1) further defining and refining the 6 ACGME competencies in the context of pediatrics, 2) identifying markers of achievement and expected performance levels across the medical education continuum, and 3) identifying and developing tools that can be embraced nationally as meaningful measures of achievement.
      • Hicks P.J.
      • Schumacher D.
      • Benson B.
      • et al.
      The Pediatrics Milestones: conceptual framework, guiding principles, and approach to development.
      The Pediatrics Milestones Project Working Group has released its first iteration of Pediatrics Milestones.

      Pediatrics Milestone Project Working Group. The Pediatrics Milestone Project. Available at: https://www.abp.org/abpwebsite/publicat/milestones.pdf. Accessed March 21, 2012.

      Building on conceptual frameworks found in the medical, social science, education, and behavioral literatures, the Pediatrics Milestones describe the proposed continuum of development for 51 subcompetencies within the ACGME competency domains, with one Pediatrics Milestone document (PMD) for each subcompetency. The background section of each PMD provides a summary of the literature used to inform the developmental progression of relevant knowledge, skills, and attitudes spanning the continuum from early medical student through graduate medical education and then continuous professional development after training.
      • Hicks P.J.
      • Schumacher D.
      • Benson B.
      • et al.
      The Pediatrics Milestones: conceptual framework, guiding principles, and approach to development.
      This background provides the foundation for a series of developmental milestones (DM) that follow it. These DM provide narrative descriptions of behaviors and inferences that demonstrate advancement along the continuum of physician development. Figure 1 shows an example of the structural anatomy of the DM for clinical reasoning. The associated full PMD, comprised of its background, DM, and related references, is shown online in Appendix A.
      Figure thumbnail gr1
      Figure 1Example of developmental milestones from the Pediatrics Milestones for clinical reasoning.
      To accomplish the goal of advancing CBME through use of the Pediatrics Milestones as road maps for learning and assessment, learner and faculty buy-in and understanding are critical.
      • Hicks P.J.
      • Englander R.
      • Schumacher D.J.
      • et al.
      Pediatrics Milestone Project: Next Steps Toward Meaningful Outcomes.
      Thus, the purpose of this study was to determine how pediatric residents understand, process, and respond to a representative sample of the PMDs.

      Methods

      Setting

      This study was conducted with residents from the pediatric residency training programs at Cincinnati Children's Hospital Medical Center (CCHMC), in Cincinnati, Ohio, and Dayton Children's Hospital (DCH) in Dayton, Ohio. The respective program sizes are approximately 55 residents (CCHMC) and 16 residents (DCH) per year.

      Participants

      We recruited 48 categorical pediatric residents for this study, 16 from each postgraduate year of a 3-year pediatric residency. Six different residents (2 from each postgraduate training year) were interviewed for each of 8 PMDs studied.
      To recruit participants, we first randomized the names of all categorical pediatric residents at CCHMC and DCH for each year of training at each institution. We then sent recruitment e-mails to residents on the randomized lists until 16 residents from each postgraduate year agreed to participate, while also ensuring half the total recruited residents were from each residency program. Although we began to recruit and interview second- and third-year residents (Pediatric Level 2 [PL-2] and Pediatric Level 3 [PL-3], respectively) in July 2010, we waited until November 2010 to recruit and interview interns (Pediatric Level 1 [PL-1]) to allow transition time to residency. Therefore, PL-1 recruitment was considered separately from that of PL-2 and PL-3. All interviews were completed by February 2011. Recruitment e-mails were sent to residents in the order determined by the randomization, regardless of their likely availability on the basis of rotation or vacation schedules. A total of 97 residents were contacted to recruit 48 participants. For their time, participants were given gift cards totaling $50.

      Study Design and Data Collection

      This study used cognitive interviews
      • Graneheim U.H.
      • Lundman B.
      Qualitative content analysis in nursing research: concepts, procedures, and measures to achieve trustworthiness.

      Willis GB. Cognitive Interviewing: A “How To” Guide, 1999. Available at: http://appliedresearch.cancer.gov/areas/cognitive/interview.pdf. Accessed March 21, 2012.

      • Willis G.B.
      Cognitive Interviewing: A Tool for Improving Questionnaire Design.
      to investigate how pediatric residents understand, process, and respond to the Pediatrics Milestones.
      • Willis G.B.
      Cognitive Interviewing: A Tool for Improving Questionnaire Design.
      Classically used in the development of survey instruments, cognitive interviews can be used in the development of most any written educational material. Cognitive interviews utilize the techniques of think-alouds and verbal probes. Think-alouds are nonleading, open-ended questions or statements asking participants to describe what they are thinking and sometimes how they are feeling in response to something they have read. Verbal probes are directed questions, often after a think-aloud, aimed at gaining more information about a participant's response.

      Willis GB. Cognitive Interviewing: A “How To” Guide, 1999. Available at: http://appliedresearch.cancer.gov/areas/cognitive/interview.pdf. Accessed March 21, 2012.

      The primary researcher (DS) conducted all interviews individually. Interviews were performed in a quiet room, ranged from 15 to 60 minutes in length, and were audio recorded. Each participant reviewed the background and then the accompanying DM for one PMD, with the interviewer asking questions after each part. A scripted protocol (Appendix B, available online) allowed interviews to be standardized across participants. Additionally, spontaneous verbal probing was used as needed to ensure that participants' comments were captured accurately. Interview questions asked participants: 1) to describe what they were thinking as they read both the background and the DM, 2) to summarize both the background and DM in their own words, 3) to place various levels of learners at the various DM performance levels, 4) to describe what they would change to make the background and DM more understandable, and 5) to describe what, if anything, helped to make them understandable. In addition to these more general questions aimed at investigating understanding and interpretation of the PMDs, we also asked participants one directed question: how they would feel if assessed as being at a performance level in the middle of the continuum of DM.
      Eight PMDs were chosen for study on the basis of their representative range of length, complexity, competency domain, and primary author. Although each PMD underwent a group vetting process, the individual writing styles of the primary authors were preserved in the final products. Identification of primary author was removed from the documents before the study. The PMDs studied are shown in Figure 2 and include those for clinical reasoning, handovers, supervision (studying the DM for the supervisee only), interprofessional communication, self-directed learning, care coordination, advocacy, and managing ambiguity. PMDs were assigned to participants in a predetermined order, generally filling interview dates for one before moving to the next. PL-1 participant assignments were done separately from PL-2 and PL-3 assignments according to the timing of PL-1 recruitment.
      This study received exempt status after review by the institutional review boards at both study sites. Written informed consent was obtained for all interviews.

      Data Analysis

      After interviews were completed, audio recordings were transcribed verbatim. Primary coding of interview transcripts and notes from the interviews was done by the researcher who conducted all interviews and was closest to the data (DS), creating inductive codes that were then grouped into themes. HyperRESEARCH version 3.0.1 (ResearchWare Inc, Randolph, Mass) was used for coding. For substantiation, codes were expanded and revised as appropriate as coding was undertaken to ensure new data was captured as encountered. To further substantiate codes and themes, 3 additional authors (MLS, JS, MAP), not part of the Pediatrics Milestones Project or the primary data collection for this study, each independently coded a subset of the interview transcripts. Two of these individuals (MLS, JS) coded a predetermined 25% of the interviews, and one (MAP) coded 15% before she thought that theme saturation was met.

      Results

      Four themes emerged through coding and analysis of the PMDs studied: 1) participants' degree of understanding is sufficient, often deep; 2) the etiology of participants' understanding is rooted in their experiences; 3) there are qualities of the PMDs that may contribute to or detract from understanding; and 4) participants apply their understanding by noting the PMDs describe a developmental progression that can provide a road map for learning. Additionally, it was learned that residents are generally comfortable being placed in the middle of a series of DM. Finally, 2 minor themes focusing on interest and practicality were also identified. No difference in themes was identified between study sites or training levels.
      In retrospect, saturation of major themes was achieved by the first 12 interviews. However, coding was continued for all 48 interviews to be sure these themes would not change after all PMDs chosen for study as well as sufficient numbers of residents from all training levels were included.

      Theme 1: Degree of Understanding is Sufficient, Often Deep

      Participants summarized the PMDs well in their own words, and all residents were able to use the DM to place various levels of learners (medical students, interns, and senior residents) they have worked with at levels they felt to be most appropriate. Although these abilities demonstrate at least sufficient understanding of the PMDs, a number of participants also made comments that demonstrated deep understanding. These comments include most participants noting that performance in the subcompetencies represented by the PMDs is context-dependent, learner-dependent, or dependent on both these variables. This is demonstrated in the following response to choosing a DM level for a senior resident:I think it depends on the patient and it depends on the senior resident. … I know senior residents that would definitely be at the fourth developmental milestone, but there's some that are kind of in between as well, depending on how comfortable they are with the disease process. If it's bread and butter pediatrics, they are going to be at the fourth developmental milestone … but if it's something like a weird heme/onc case or something like that, they're not going to be like it's definitely this, we are definitely going to do this. (Interview 18, Clinical Reasoning, PL-1)
      Noting learner-dependence, another participant noted:We all develop at a different rate, and some people are going to take a little bit longer I feel like … to kind of reach that point. (Interview 46, Ambiguity, PL-3)
      Also demonstrating deeper understanding, a number of participants noted that the DM are a continuum in which learners can fall between DM levels. For example, one participant noted:I do feel like it's a continuum even within these five [DM]. I think that someone could be somewhere between two and three and somewhere between four and five very easily. (Interview 14, Supervision, PL-2)
      Further demonstrating deeper understanding, a number of participants noted that the individual elements of the DM (Figure 1) may not track together (ie, a resident may be on one level for one element and another level for another element), as in the participant who noted:You did maybe this level here … and this level here for part of it, and so you are in between two of the milestones. (Interview 21, Care Coordination, PL-2)
      In response to where he would place medical students on the DM, another participant noted:I would probably place them at the second … and I might even, with some aspects of it, place it at the third [developmental milestone level]. (Interview 31, Care Coordination, PL-2)

      Theme 2: Etiology of Understanding is Rooted in Experiences

      All participants used their own experiences to place various levels of learners at the DM levels they felt to be most appropriate, and many participants noted using their personal experiences in understanding the background as well.
      In response to being asked what he was thinking when he read the background, one participant noted:I was thinking about my own experience, first as an intern, then as a senior resident during second and third year, and sort of specific examples that I ran across. (Interview 16, Supervision, PL-3)
      Another participant noted:I kind of think back to what I was like as a medical student and how I have kind of progressed through residency, and kind of like, oh yeah, I remember when I did that … As a resident, I am like, oh, our third year medical students always do that. (Interview 25, Interprofessional Communication, PL-3)
      Additionally, most participants felt that experience was important to advancement through the DM. As an example, one participant noted:To hit that fifth level [of five DM], you need to be very, very … experienced and have had to figure out a lot of problems before to get to that point. (Interview 10, Self-directed Learner, PL-3)
      In response to discussing why she places senior residents where she did on the DM, another participant noted:Some of it has to do with having more experience and being able to say this is not normal or this is different than the other ones, the other situations like this that I have seen, and I want to know why … You're not so bogged down in “this is my first month being a doctor and I'm just so frazzled that I have to order flushes and I'm not thinking about the whole picture.” (Interview 13, Self-directed Learner, PL-3)

      Theme 3: Certain Milestones Document Qualities Contribute Toward or Detract From Understanding

      A minority of participants noted positive or negative factors related to the content of the PMDs they reviewed. This included a small number of participants noting they either liked or disliked the reference of literature in the background. There were also comparable mixed reports for terms either being or not being used or defined optimally, as demonstrated in these opposing comments:The word numeracy … I think that whole concept is a little bit vague and a little bit hard to understand. (Interview 40, Ambiguity Milestones, PL-2)I think the vocabulary is easy to follow and easy to know what it means. (Interview 18, Clinical Reasoning Milestone, PL-1)
      Additionally, a small number of participants noted that concepts in the PMDs were either well-defined or not well-defined, as evidenced in these opposing comments:[In response to what language she would change to make it more understandable:] Maybe explaining the recognition of cultural issues … like what are they talking about? Are you talking about people that have cultural beliefs that keep them from wanting to transfuse their child or something? That's a very rare situation, so maybe defining that into more everyday terms or giving examples or something. (Interview 28, Care Coordination Milestone, PL-3)I think the first part that is talking about the communicative competence—where it breaks it down into grammatical, and sociolinguistic, and strategy—that's pretty straight forward, and I like how they give examples of that. (Interview 38, Interprofessional Communication, PL-3)
      A few participants noted conflicting thoughts on the length of the background or DM, feeling they were either an appropriate length or too long. Interestingly, participants noted both that the shorter PMDs studied were too long and that the longer PMDs studied were an appropriate length.
      Finally, a few participants noted needing to reread something in the background or the DM to more fully understand it, as noted in this participant's comments after reading the background:I would have to read this part a couple times I think to really grasp it, but once I got to the table I was like “okay.” (Interview 41, Ambiguity Milestone, PL-2)
      When considering what factors contributed toward participants' understanding of the PMDs, 2 characteristics predominated. First, participants felt examples in the background, and less so in the DM, are helpful. Second, they felt that bold type for key threads across a series of DM is helpful (Figure 1).

      Major Theme 4: Application of Understanding Notes the Milestone Documents Describe a Developmental Progression That Can Serve as Road Maps for Learning

      Participants noted the developmental progression of the PMDs, most notably in the DM, with many explicitly noting the PMDs provide a road map for their learning. As an example, one participant noted the following in response to what helps make the DM understandable:You can definitely see the progression in each one of them, and I think it's pretty obvious the next steps they would like you to be able to take. (Interview 35, Advocacy Milestone, PL-1)
      In response to being asked what she was thinking as she read the DM, another participant noted:It seems like you are basically putting down like different stages of where someone would be in communication, which would give them a sense of where they are and what they are working towards or specific things they could improve on … I feel like it gives you very specific goals to work towards. (Interview 44, Interprofessional Communication Milestone, PL-1)
      Some participants also noted that the PMDs could provide a national guideline with which comparisons of residents between programs can be made.

      Specific Investigation Theme: Residents are Generally Comfortable Being Placed in the Middle of a Series of DMs

      Participants were asked how they would feel if someone assessed them as being in the middle of the continuum of DM. Their responses to this question demonstrate they are largely comfortable with this assessment. As an example, one participant noted:I would feel OK with that as long as I was … meeting expectations and that that was not necessarily where everybody else was but that's where I was expected to be at this [time]. I would want to know that I'm at least where I need to be … I'm fine with realizing that there's obviously, as an intern, a lot of room to grow … I wouldn't expect me to be at the highest level or probably even the second highest [DM level]. (Interview 45, Ambiguity, PL-1)
      Of the participants who noted they would not feel comfortable if placed there, most all self-assessed as either falling at or feeling they should fall at a more advanced DM level for their training level. This is evidenced in these 2 participants' responses:I would feel … a little disappointed because I think I fit more with [the] four[th DM level] … so I need to work on a few things and, um, look at those things and try and improve them. (Interview 41, Ambiguity, PL-2)If I was at a [level] one or two, I would think I would need serious work to work on my differential diagnosis, to work on my thinking, my, all that clinical judgment. Definitely on a lot of patients I function as a [level] three [of 5 DM], and I would be okay with that. (Interview 8, Clinical Reasoning, PL-3)
      This group of participants also tended to note that while they were not where they wanted to be or expected to be, they would appreciate the feedback, as evidenced by one participant who noted:I want brutal feedback, so initially, after I get over the three second sadness, I would actually appreciate good feedback. (Interview 9, Handovers, PL-3)

      Two Minor Themes: Interest and Practicality

      A small number of codes did not fit well into one of the main themes but also did not constitute their own main themes as a result of the paucity of their use. Most of these codes comprise 2 minor themes: 1) most participants who discussed their interest level in the PMD they reviewed noted it was interesting to them, and 2) participants had mixed views on the practicality of using the PMDs with almost the same number noting the potential barriers (eg, number, length) to using the Pediatrics Milestones as noting the importance of engaging residents in using the Pediatrics Milestones to guide their learning.

      Discussion

      This study found that pediatric residents who reviewed a representative subset of the Pediatrics Milestones used their own experiences to demonstrate a sufficient, often deep, understanding of the PMDs they reviewed. Some participants noted qualities that facilitate or detract from understanding. Perhaps most notably, participants applied their understanding by noting the PMDs can describe a developmental progression that can serve as a much needed road map for learning.
      Formative feedback that learners understand is a foundational requisite to achieving competency-based medical education and assessment.
      • Carraccio C.
      • Wolfsthal S.D.
      • Englander R.
      • et al.
      Shifting paradigms: from Flexner to competencies.
      Thus, it is critical for the Pediatrics Milestones to be understandable and meaningful to learners if they are to be used with them. This study provides initial evidence for this (theme 1).
      Although participants used their own experiences to understand the Pediatrics Milestones (theme 2), the amount of experience required for sufficient understanding is less clear. To this end, it is perhaps reassuring to observe that participants drew on their own experiences as well as their experiences working with those at more advanced levels to understand the more advanced DM levels. Thus, earlier learners who have at least some experience with supervisors may not be as disadvantaged as this theme may suggest. At the very least, this study suggests that interns at least 3 months into their intern year (those whom we studied) have sufficient experiences to inform a frame of reference for use in understanding the PMDs.
      Focusing on what facilitates or detracts from understanding, some participants offered either positive or negative sentiments around the use of terms and concepts, the references to the literature, and the length of the PMD they reviewed (theme 3). Although these comments were from a minority, the impact of individuals with similar sentiments is unclear if applied to the broader pediatric resident population. Would those with either positive or negative thoughts have more effect on group thinking or would both of these groups remain a minority that has little influence on the whole? Would trainees from some residency programs have more positive sentiments or more negative sentiments based on the culture of their program? Future study with residents from more institutions and with group interviews could help to explore these unanswered questions.
      Perhaps most notably, this study suggests that the Pediatrics Milestones may have the potential to provide a much needed road map for learning (theme 4). Too often in medical education we fail to clearly communicate our expectations to learners. Without these clearly stated and objective expectations, development may be left to chance. Although developmental progression may still occur in these situations, it may not occur as efficiently or effectively. Thus, learning that residents think the Pediatrics Milestones can provide a road map for their learning provides much hope for addressing an important need in optimizing medical education and assessment.
      The approach to the Pediatrics Milestones and competency-based medical education and assessment in general represents a paradigm shift from current efforts. The literature for traditional assessment efforts is replete with evidence that faculty inflate ratings of learners,
      • Varney A.
      • Todd C.
      • Hungle S.
      • et al.
      Description of a developmental criterion-referenced assessment for promoting competence in internal medicine residents.
      • Roman B.J.B.
      • Trevino J.
      An approach to address grade inflation in a psychiatry clerkship.
      • Speer A.J.
      • Solomon D.J.
      • Fincher R.E.
      Grade inflation in internal medicine clerkships: results of a national survey.
      with assessment often being in the eye of the beholder
      • Kogan J.R.
      • Conforti L.
      • Bernabeo E.
      • et al.
      Opening the black box of clinical skills assessment via observation: a conceptual model.
      This could create the false impression among learners that being assessed where one falls represents poor performance if the rating is not inflated. Narrative descriptions of behaviors, such as those set forth in the PMDs, leave less room for interpretation and provide more hope for accurate assessments. Although this should be viewed positively, it is possible these descriptions could be perceived as a threat to learners accustomed to inflated ratings with current tools when they fall at an earlier developmental level. Although we anticipated this could be true for the residents in our study for the reasons discussed here, the data did not support this hypothesis. Rather, participants in our study were comfortable being assessed in the middle of the DM continuum as they felt their own behavior and/or that of their colleagues often matched the behavioral descriptors in this range on the continuum. They often saw only senior residents, fellows, or attendings falling at the most advanced DM level(s), matching the intent of the Pediatrics Milestones Project Working Group to create DM that span the continuum from novice to master.
      • Hicks P.J.
      • Schumacher D.
      • Benson B.
      • et al.
      The Pediatrics Milestones: conceptual framework, guiding principles, and approach to development.
      This finding provides hope that using the Pediatrics Milestones will allow supervisors to assign a level to a learner that matches the learner's actual performance, no matter how early in development, without fear of damaging resident self-esteem. The only potential threat raised in our interviews is instances when residents' perceived levels do not match their supervisor-assigned levels. However, even in these situations, our participants note the effectiveness of this feedback in motivating them to improve or discuss this assignment further with the supervisor.
      Finally, the themes in this study raise the question of whether the Pediatrics Milestones may provide a framework that is useful to learners in other specialties as well, as the subcompetencies they address speak to physician development in general. If the Pediatrics Milestones were applied to other specialties, however, it is unclear whether the concept of a developmental progression would resonate with them as it does for pediatric residents who are so familiar with the concept of child development.

      Assumptions and Limitations

      In conducting this study, a few key assumptions were made. First, we assumed that cognitive interviewing is useful for determining how pediatric residents understand, interpret, and respond to the PMDs in a nonbiased manner. Given that cognitive interviewing is a valid method for determining interpretation and response to written information, this assumption is felt to be appropriate. However, it is possible that residents may not have felt comfortable speaking freely as the interview was one-on-one with the researcher. It is also possible that residents altered their comments because the interviewer, as well as the site PI for DCH, are members of the Pediatrics Milestone Project. We think that such alterations of residents' comments are a less likely confounder as we gathered both positive and negative responses and participants spoke freely about perceived short-comings in their training environments. Furthermore, our study focus was not on whether residents liked the PMDs. Additionally, although the Pediatrics Milestone Project has received much attention recently, all interviews for this study were conducted before this time. Indeed, interviews were completed almost 1 year before the first public release of the Pediatrics Milestones. Thus, we think it is unlikely that heightened discussions about the Pediatrics Milestone Project introduced bias into participant responses. Indeed, multiple participants commented that they first learned about the Milestone Project at the time of participation in our study. Another potential limitation is the assumption that studying a representative set of 8 PMDs will yield information that is pertinent to all 51. Although the PMDs included in this study were chosen as a representative sample, it is possible that further themes would be identified if additional PMDs were studied. However, it should be noted that theme saturation was achieved within the current PMDs studied and that additional global themes were not identified after coding the first few PMDs studied. Third, it is assumed that pediatric residents from 2 Midwest pediatric residency programs will represent the views of residents at other programs and pediatric residents in general. Future study should include residents from other areas of the country as well as residents from other programs. Fourth, it is assumed that categorical pediatric residents will represent the views of those in combined training programs (ie, medicine/pediatrics and child psychiatry/adult psychiatry/pediatrics), for whom the Pediatrics Milestones will also be used. Finally, as noted previously, pediatrics residents are accustomed to thinking about the developmental milestones of children. Thus, they may more readily understand the concept of a developmental trajectory when compared to residents of other specialties, potentially affecting the use of the Pediatrics Milestone framework in other specialties.

      Conclusion

      Since the advent of CBME, much effort and focus has been placed on realizing its promise in our medical education settings.
      • Carraccio C.
      • Wolfsthal S.D.
      • Englander R.
      • et al.
      Shifting paradigms: from Flexner to competencies.
      • Harris P.
      • Snell L.
      • Talbot M.
      • et al.
      International CBME Collaborators
      Competency-based medical education: implications for undergraduate programs.
      • Frank J.R.
      • Snell L.
      • Cate O.
      • et al.
      Competency-based medical education: theory to practice.
      • Holmboe E.S.
      • Ward D.S.
      • Reznick R.K.
      • et al.
      Faculty development in assessment: the missing link in competency-based medical education.
      • Holmboe E.
      • Sherbino J.
      • Long D.M.
      • et al.
      International CBME Collaborators
      The role of assessment in competency-based medical education.
      • Talbot M.
      In defence of a human conversation in medical vocational education: a discussion from the United Kingdom.
      With challenges to achieving this paradigm shift, the results to date have been mixed.
      • Snell L.
      • Frank J.
      Competencies, the tea bag model, and the end of time.
      • Lurie S.J.
      • Mooney C.J.
      • Lyness J.M.
      Meaurement of the general competencies of the accreditation council for graduate medical education: a systematic review.
      The Milestone Project hopes to take a significant step toward achieving the promise of CBME. As a central tenet of CBME, the Pediatric Milestones Project strives to place the learner at the center of its work. This study provides initial evidence for the Pediatrics Milestones as learner-centered documents that can be used for orientation, education, formative feedback, and, ultimately, for assessment.

      Acknowledgments

      We would like to acknowledge and thank the University of Cincinnati College of Medicine for a Medical Education Research Project Grant (PI: Daniel J. Schumacher, MD) to fund this study. We would also like to thank the Cincinnati Children's Hospital Medical Center Division of Emergency Medicine for additional funds to complete this project.

      Supplementary Data

      References

        • Nasca T.J.
        • Philibert I.
        • Brigham T.
        • et al.
        The next GME accreditation system—rationale and benefits.
        N Engl J Med. 2012; 366: 1051-1056
        • Hicks P.J.
        • Schumacher D.
        • Benson B.
        • et al.
        The Pediatrics Milestones: conceptual framework, guiding principles, and approach to development.
        J Grad Med Educ. 2010; 2: 410-418
        • Hicks P.J.
        • Englander R.
        • Schumacher D.J.
        • et al.
        Pediatrics Milestone Project: Next Steps Toward Meaningful Outcomes.
        J Grad Med Educ. 2010; 2: 577-584
      1. Pediatrics Milestone Project Working Group. The Pediatrics Milestone Project. Available at: https://www.abp.org/abpwebsite/publicat/milestones.pdf. Accessed March 21, 2012.

        • Graneheim U.H.
        • Lundman B.
        Qualitative content analysis in nursing research: concepts, procedures, and measures to achieve trustworthiness.
        Nurse Educ Today. 2004; 24: 105-112
      2. Willis GB. Cognitive Interviewing: A “How To” Guide, 1999. Available at: http://appliedresearch.cancer.gov/areas/cognitive/interview.pdf. Accessed March 21, 2012.

        • Willis G.B.
        Cognitive Interviewing: A Tool for Improving Questionnaire Design.
        Sage, Thousand Oaks, Calif2005
        • Carraccio C.
        • Wolfsthal S.D.
        • Englander R.
        • et al.
        Shifting paradigms: from Flexner to competencies.
        Acad Med. 2002; 77: 361-367
        • Varney A.
        • Todd C.
        • Hungle S.
        • et al.
        Description of a developmental criterion-referenced assessment for promoting competence in internal medicine residents.
        J Grad Med Educ. 2009; 1: 73-81
        • Roman B.J.B.
        • Trevino J.
        An approach to address grade inflation in a psychiatry clerkship.
        Acad Psychiatry. 2006; 30: 110-115
        • Speer A.J.
        • Solomon D.J.
        • Fincher R.E.
        Grade inflation in internal medicine clerkships: results of a national survey.
        Teach Learn Med. 2000; 12: 112-116
        • Kogan J.R.
        • Conforti L.
        • Bernabeo E.
        • et al.
        Opening the black box of clinical skills assessment via observation: a conceptual model.
        Med Educ. 2011; 45: 1048-1060
        • Harris P.
        • Snell L.
        • Talbot M.
        • et al.
        • International CBME Collaborators
        Competency-based medical education: implications for undergraduate programs.
        Med Teach. 2010; 32: 646-650
        • Frank J.R.
        • Snell L.
        • Cate O.
        • et al.
        Competency-based medical education: theory to practice.
        Med Teach. 2010; 32: 638-645
        • Holmboe E.S.
        • Ward D.S.
        • Reznick R.K.
        • et al.
        Faculty development in assessment: the missing link in competency-based medical education.
        Acad Med. 2011; 86: 460-467
        • Holmboe E.
        • Sherbino J.
        • Long D.M.
        • et al.
        • International CBME Collaborators
        The role of assessment in competency-based medical education.
        Med Teach. 2010; 32: 676-682
        • Talbot M.
        In defence of a human conversation in medical vocational education: a discussion from the United Kingdom.
        J Vocat Educ Train. 2001; 53: 663-676
        • Snell L.
        • Frank J.
        Competencies, the tea bag model, and the end of time.
        Med Teach. 2010; 32: 629-630
        • Lurie S.J.
        • Mooney C.J.
        • Lyness J.M.
        Meaurement of the general competencies of the accreditation council for graduate medical education: a systematic review.
        Acad Med. 2009; 84: 301-309