Advertisement

The Pediatrics Milestones: Pursuit of a National System of Workplace-Based Assessment Through Key Stakeholder Collaboration

      The Pediatrics Milestones Project is now in its fourth year since the Accreditation Council for Graduate Medical Education’s (ACGME) original charge to develop milestones-based assessment for the workplace. Collaborative efforts have created the foundation to evaluate a national assessment system to ensure the use of meaningful assessment instruments and to facilitate the implementation of associated faculty development and curricular changes. Building on the past, we look enthusiastically to embark on large-scale research collaboration, necessary to validate the milestones approach to assessment.

      Collaboration and Organizational Transformation

      Collaborative relationships with the potential for organizational transformation usually require a long period during which the parties develop expertise while directing their efforts toward a shared goal. If the growth and change align with organizational core values, and if the parties commit to continuous collaboration, the outcomes realized are often greater than the sum of what each individual group’s work could produce independently. In this place of sustained synergy, innovation can be optimized and transformative change can occur.

      Long-standing Collaboration

      The American Board of Pediatrics (ABP) and the Association of Pediatric Program Directors (APPD) have had a fruitful and close working relationship for many years. The Program Directors Committee (PDC) of the ABP has had active APPD members on the committee. This committee’s work product contributed to the pediatrics community through publications such as Teaching and Assessing Professionalism and Assessment in Graduate Medical Education: A Primer for Program Directors. This committee’s function is to provide communication between the organizations and, when called for, to offer responses to current or anticipated needs. The collaboration between the ABP and APPD set the stage for the Pediatrics Milestones Project, an initiative of the ACGME and the ABP. The Pediatrics Milestones Working Group, assembled and led by Carol Carraccio, included many APPD members and PDC members; the Pediatrics Milestones Project

      Benson BJ, Burke A, Carraccio C et al. The Pediatrics Milestones Project. A joint initiative of the Accreditation Council for Graduate Medical Education and the American Board of Pediatrics. Available at: http://www.acgme.org/acWebsite/RRC_320/320_PedsMilestonesProject.pdf and http://www.abp.org/abpwebsite/publicat/milestones.pdf2012. Accessed January, 2012.

      represents yet another fruit of this collaboration.

      Organizational Development Sets the Stage for Optimizing Collaborative Work

      Association of Pediatric Program Directors

      APPD’s Longitudinal Educational Research Network (LEARN) was developed as part of the APPD 2006 strategic plan, with the mission “to conduct meaningful educational research that advances the training of future Pediatricians by developing and promoting participation and collaboration in research by program directors for the purpose of improving the health and well-being of children.”

      Association of Pediatric Program Directors (APPD); Longitudinal Educational Assessment Research Network (LEARN). APPD LEARN mission statement. Available at: http://learn.appd.org. Accessed October 22, 2013.

      APPD’s vision is to advance exemplary pediatric education through collaborative educational research by Pediatric program directors. APPD LEARN pursues its mission through the following interconnected activities:
      • Managing a collaborative research network of pediatric programs working together to conduct multisite studies of educational methods and instruments.
      • Maintaining an online repository of educational research study materials, raw data, and findings for dissemination to APPD members and collaborators.
      • Promoting learning opportunities to enhance educational research participation and scholarship by program directors.
      • Providing expert consultation for research conducted within APPD LEARN.
      • Communicating regularly with the APPD membership and the larger medical education community about activities, opportunities, and outcomes.
      • Exploring, conducting, and coordinating research with other organizations on initiatives across a continuum of medical and nonmedical education.
      Support for APPD LEARN has come from the American Board of Pediatrics Foundation and the APPD membership, allowing the infrastructure to be built and the collaboration sustained.

      American Board of Pediatrics

      At the same time, organizational changes at the ABP were occurring with an increased commitment to competency-based medical education. In 2011, the ABP created a new position for vice president of competency-based medical education. The ABP was responding to the need to advance competency-based medical education by addressing the major gap in assessment of competence. The need for assessment to better inform the verification of clinical competence called for instruments that could provide the evidence about competence and for advancement decisions.

      National Board of Medical Examiners

      The National Board of Medical Examiners (NBME) is best known for cosponsoring the United States Medical Licensing Examination. The long-standing mission of the NBME is to “protect the health of the public through state of the art assessment of health professionals.” Although the APPD was developing its research network and the ABP was directing resources toward competency-based medical education, the NBME continued with research beyond the traditional examination setting, including faculty volunteers from the medical education community in efforts to develop workplace-based assessment, focusing on the Assessment of Professional Behaviors.

      National Board of Medical Examiners (NBME). Assessment of professional behaviors. Available at: http://www.nbme.org/apb/. Accessed October 22, 2013.

      Common Streams of Development Lead to New Collaboration

      In 2011, at the cusp of the release of the Pediatrics Milestones Project Working Group’s first iteration of the Pediatrics Milestones Project,

      Benson BJ, Burke A, Carraccio C et al. The Pediatrics Milestones Project. A joint initiative of the Accreditation Council for Graduate Medical Education and the American Board of Pediatrics. Available at: http://www.acgme.org/acWebsite/RRC_320/320_PedsMilestonesProject.pdf and http://www.abp.org/abpwebsite/publicat/milestones.pdf2012. Accessed January, 2012.

      representatives of the ABP, the NBME, and the APPD gathered to discuss the need for further study of the pediatrics milestones. Although the working group provided a conceptual framework for each of the pediatrics milestones, the corresponding levels of achievement of outcomes of individual competencies were untested.
      • Hicks P.J.
      • Schumacher D.J.
      • Benson B.J.
      • et al.
      The Pediatrics Milestones: conceptual framework, guiding principles, and approach to development.
      APPD leadership met with NBME leadership to discuss ways in which the pediatrics milestones could be studied through APPD LEARN. The NBME, with its expertise in assessment, and the newly formed APPD LEARN, with its broad network of pediatrics residency programs eager to conduct assessment research, resulted in a natural synergy that seemed obvious.
      After a series of meetings, the 2 organizations launched the Pediatrics Milestones Assessment Pilot,

      Association of Pediatric Program Directors (APPD); Longitudinal Educational Assessment Research Network (LEARN); National Board of Medical Examiners (NBME). APPD-LEARN NBME pediatrics milestones assessment pilot. Available at: http://learn.appd.org/.

      a study exploring the feasibility of a series of pediatrics milestones–based instruments designed to produce evidence to inform the decision about subintern and intern readiness to serve on the inpatient pediatrics wards. Instruments developed for this pilot included a structured clinical observation (SCO)
      • Lane J.L.
      • Gottlieb R.P.
      Structured clinical observations: a method to teach clinical skills with limited time and financial resources.
      survey for rounds, a SCO for history taking, and a multisource, multiencounter direct observation survey. Access to the instruments was provided through an application that linked the Web-based front end to any mobile device, increasing the ease of rater scoring and offering scoring at the point of care and offering feedback providers the opportunity to access completed SCOs for immediate feedback to the observee. The success of this project required the collaboration and expertise of the pediatric medical education community, APPD LEARN, and the NBME Center for Innovation

      National Board of Medical Examiners (NBME) Center for Innovation. Available at: http://www.nbme.org/research/ci.html.

      ; support for this project came from the NBME, ABP, and APPD.
      Continued development of each of the key collaborator’s expertise is critical for the ongoing work of this collaboration. Much remains to be learned about assessment of physicians in the workplace. In the future, correlation with patient outcomes will also be critically important. The collaboration of the ABP, NBME, and APPD offers the unique opportunity to create a national assessment system that can serve as a prototype for other specialties.

      Next Steps

      To develop an assessment system that provides strong evidence for progression decisions, efficiency, and administrative ease, ongoing collaboration and great resources are needed. Each organization will need to continue to increase its expertise and add to its portfolio while sharing new ideas and collaborating in work toward larger common goals. The continued commitment of APPD members to conduct research through APPD LEARN is critical so that the final product can provide the utility
      • VanderVleuten C.P.M.
      The assessment of professional competence: developments, research and practical implications.
      needed by programs. As in any change, cultural shifts and associated new activities are called for; sustaining best practice of new assessment methods will require going beyond research and implementation. Success of the pediatrics milestones assessment system will require ongoing commitment, trust, and deliberative collaboration, championed by the leaders of each organization and by the academic institutional leadership positioned to support best practices.

      References

      1. ABP. ABP programs directors resources. Available at: http://www.abp.org/ABPWebStatic/?anticache=0.016362170223146677-murl%3D%2FABPWebStatic%2Fprogdirector.html%26surl%3D%2Fabpwebsite%2Fprogramdirectors%2Fprogramdirectors.htm.

      2. Benson BJ, Burke A, Carraccio C et al. The Pediatrics Milestones Project. A joint initiative of the Accreditation Council for Graduate Medical Education and the American Board of Pediatrics. Available at: http://www.acgme.org/acWebsite/RRC_320/320_PedsMilestonesProject.pdf and http://www.abp.org/abpwebsite/publicat/milestones.pdf2012. Accessed January, 2012.

      3. Association of Pediatric Program Directors (APPD); Longitudinal Educational Assessment Research Network (LEARN). APPD LEARN mission statement. Available at: http://learn.appd.org. Accessed October 22, 2013.

      4. National Board of Medical Examiners (NBME). Assessment of professional behaviors. Available at: http://www.nbme.org/apb/. Accessed October 22, 2013.

        • Hicks P.J.
        • Schumacher D.J.
        • Benson B.J.
        • et al.
        The Pediatrics Milestones: conceptual framework, guiding principles, and approach to development.
        J Grad Med Educ. 2010; 2: 410-418
      5. Association of Pediatric Program Directors (APPD); Longitudinal Educational Assessment Research Network (LEARN); National Board of Medical Examiners (NBME). APPD-LEARN NBME pediatrics milestones assessment pilot. Available at: http://learn.appd.org/.

        • Lane J.L.
        • Gottlieb R.P.
        Structured clinical observations: a method to teach clinical skills with limited time and financial resources.
        Pediatrics. 2000; 105: 973-977
      6. National Board of Medical Examiners (NBME) Center for Innovation. Available at: http://www.nbme.org/research/ci.html.

        • VanderVleuten C.P.M.
        The assessment of professional competence: developments, research and practical implications.
        Adv Health Sci Educ. 1996; 1: 54