Advertisement

Building Bridges Between Silos: An Outcomes-Logic Model for a Multidisciplinary, Subspecialty Fellowship Education Program

Published:September 02, 2015DOI:https://doi.org/10.1016/j.acap.2015.08.010
      Graduate medical education program directors face new challenges as the Accreditation Council of Graduate Medical Education (ACGME) modifies curriculum standards. Current standards require “a formally-structured educational program in clinical and basic sciences related to the subspecialty,” and these requirements involve training in areas beyond direct patient care.
      Accreditation Council for Graduate Medical Education
      ACGME program requirements for graduate medical education in the subspecalties of pediatrics.
      Similarly, the American Board of Pediatrics (ABP) has expanded curriculum requirements for fellowship training,

      American Board of Pediatrics. Changes in training requirements for subspecialty certification. Available at: https://www.abp.org/content/general-criteria-subspecialty-certification. Accessed August 14, 2010.

      and the Federation of Pediatric Organizations has added that training in scholarship, core competencies, and skills in lifelong learning and teaching also should be incorporated.
      Federation of Pediatric Organizations
      Policy statement—Pediatric fellowship training.
      The Council of Pediatric Subspecialties was formed in part to address the extent to which pediatrics fellowship programs provide comprehensive training for successfully transitioning to subspecialty academic careers. It has yet to respond to the substantial need for shared curricula common to pediatric fellowships.
      • Mink R.
      • Norwood V.
      • Degnon L.
      • et al.
      Council of Pediatric Subspecialties (CoPS): the first five years.
      Some pediatric subspecialty organizations have collaborated across institutions to develop subspecialty-specific curricula.
      • Gupta M.
      • Ringer S.
      • Tess A.
      • et al.
      Developing a quality and safety curriculum for fellows: lessons learned from a neonatology fellowship program.
      • Martinez J.A.
      • Koyama T.
      • Acra S.
      • et al.
      Nutrition education for pediatric gastroenterology, hepatology, and nutrition fellows: survey of NASPGHAN fellowship training programs.
      • Hastings C.
      • Wechsler D.S.
      • Stine K.C.
      • et al.
      Consensus on a core curriculum in American training programs in pediatric hematology-oncology: a report from the ASPHO Training Committee.
      However, their priorities are specific, rather than general, in scope.
      Adding to the challenge is the paucity of examples in the literature of methods to develop, implement, and evaluate a comprehensive fellowship curriculum within an institution or department. We describe the initial steps we took to address these challenges and the use of a logic model to create Fellows' College (FC), a centralized educational program for subspecialty trainees in the Department of Pediatrics at Baylor College of Medicine (BCM), Houston, Texas.
      Ethical approval was obtained from the Baylor College of Medicine institutional review board (approval H32258).

      Situation

      In 2010, several subspecialty fellowship programs received citations related to deficiencies in core curriculum requirements, inadequate practice-based learning and improvement opportunities, and lack of individualized development plans for trainees. At that time, each program functioned independently. Recognizing that subspecialty silos contributed to these deficiencies and variations in duration of accreditation cycles, one of us (JRC) suggested creating a centralized educational program for all pediatric fellows as a mechanism to address these challenges. The goals were to enhance subspecialty education for fellows and to support program directors and coordinators. Department leadership supported the vision and provided resources for our planning process.

      Planning

      We first conducted an in-depth review of the literature and the ACGME and ABP requirements. We then performed a needs assessment to identify common educational deficiencies in our fellowship programs. Valuing models of best practices in education, we identified and consulted with an established centralized educational program at the University of California, San Francisco. Departmental leadership provided funds to bring the director of this program to BCM as a visiting professor. Insights gained from that consultation helped us define an innovative leadership structure and select the best model for developing our program.

      Defining a Leadership Structure

      The leadership structure of FC was based on the principles outlined in Jim Collins's monograph Good to Great and the Social Sectors.
      • Collins J.
      Good to Great and the Social Sectors.
      We formed a steering committee consisting of junior and senior faculty members from 8 pediatric subspecialties to serve as an advisory board. With broad representation, we were able to “get the right people on the bus”
      • Collins J.
      Good to Great and the Social Sectors.
      and identify and eliminate deficiencies in our subspecialty training programs.

      Selecting and Drafting a Logic Model

      A logic model is a “systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan to do, and the changes or results you hope to achieve.”
      W. K. Kellogg Foundation
      Using Logic Models to Bring Together Planning, Evaluation, and Action: Logic Model Development Guide.
      These models are helpful tools for guiding the development, implementation, and evaluation of a multifaceted program.
      • Armstrong E.G.
      • Barsion S.J.
      Using an outcomes-logic model approach to evaluate a faculty development program for medical educators.
      The 3 approaches for a logic model are: 1) theory (conceptual), 2) outcomes, and 3) activities (applied). To ensure a shared understanding of and focus on the goals of FC, we chose to implement an outcomes-logic model.
      An outcomes-logic model includes inputs, outputs, and outcomes (Fig). The inputs consist of the resources available to the program (items or people needed to make the program happen). The outputs are activities/strategies (how the program uses the resources to achieve its mission) and the intended participants. Usually outcomes are grouped into 3 separate categories on the basis of length of time: 1) immediate results (<1 year), 2) intermediate benefits (1–3 years) for fellowship programs and participants, and 3) the impact (eg, changes or benefits) of the program on the organization (3–5 years).
      Figure thumbnail gr1
      FigureIllustration of an abbreviated version of the BCM Fellows' College Logic Model.
      We organized a half-day retreat for 16 key stakeholders (eg, vice chair of education, core pediatric program director, departmental educational leaders, and selected influential fellowship program directors) and leaders of our FC to develop our outcomes-logic model. Before the meeting, we gave the participants 3 items: 1) a paper from the W. K. Kellogg Foundation describing the development of a logic model
      W. K. Kellogg Foundation
      Using Logic Models to Bring Together Planning, Evaluation, and Action: Logic Model Development Guide.
      ; 2) an article by Armstrong and Barsion
      • Armstrong E.G.
      • Barsion S.J.
      Using an outcomes-logic model approach to evaluate a faculty development program for medical educators.
      describing how they used an outcomes-logic model to evaluate a faculty development program; and 3) an example of an outcomes-logic model from another BCM program.
      Retreat participants received a blank outcomes-logic model template as described above. The preretreat assignment was to identify inputs (resources), proposed activities, participants, and desired outcomes (immediate, intermediate, and impact) considered to be goals that FC should accomplish. The retreat was facilitated by one of the authors (TLT), who had developed the example outcomes-logic model given to the participants. Stating proposed outcomes up front was instrumental in determining the content to include and the parameters to evaluate, as well as in developing a structured proposal to present to the departmental leadership. The Figure is an abbreviated visual representation of the complete outcomes-logic model developed and presented to the departmental leadership.

      Inputs

      Educational leaders in the department agreed to serve as advisors and mentors for the directors of FC. This support was vital to the implementation of the program. We applied for and were awarded a competitive grant from Texas Children's Hospital, which provided $80,000 over 2 years to implement the program. Budget categories included: travel and registration for faculty to attend ACGME and Association of Pediatric Program Directors meetings, FC administrative salary support, and meals for all FC sessions. We leveraged existing resources in subspecialty sections by inviting those who were leading efforts to partner with us. Examples are an academic career development workshop series developed by our pediatric pulmonology program director
      • Rama J.A.
      • Campbell J.R.
      • Balmer D.F.
      • et al.
      Investing in future pediatric subspecialists: a fellowship curriculum that prepares for the transition to academic careers.
      and an educator skills series developed by our academic general pediatrics program director. We also collaborated with leaders of a successful research-focused symposium at our institution and then offered instructive sessions on abstracts, posters, and oral presentations.

      Outputs: Activities

      Program directors and coordinators had similar needs for orientation of new fellows, but each program addressed these needs in subspecialty silos. Thus, we developed a departmental pediatric subspecialty fellowship orientation. It provided a venue for encouraging fellows to form peer networks that are important for building team skills and collaboration. We also used the orientation to establish a foundation for the academic core curriculum and conducted reflective sessions directed at goal setting and individualized professional development plans, thereby addressing a deficiency identified by ACGME site visits.
      With faculty engagement, we were able to offer a comprehensive and coordinated curriculum to all subspecialties throughout the year. Curriculum content was developed on the basis of the needs assessments results and ACGME and ABP requirements. Session topics addressed the concerns of 1 of 3 major modules (Table): 1) ACGME and ABP Core Curriculum, 2) Academic Career Development, and 3) Research and Scholarship Training. As the program progressed, we added topics such as Quality Improvement, Patient Safety, and Physician Wellness. Several faculty members proposed topics to be included in FC, as they sought a centralized home for their efforts and desired opportunities for collaboration and mentoring. We subsequently integrated 2 optional modules, Global Health and Simulation, which are important features for specific learners. Over the course of 4 years, we have expanded the content of the curriculum nearly 3-fold and now offer 40 sessions each year.
      TableFellows' College Modules
      Each session is 1 hour long unless otherwise indicated.
      • Core Curriculum (25 sessions)
        • Fellows as Educators (4 sessions)
        • Communication (4 sessions)
        • Mentors and Mentees
        • Professionalism (2 sessions)
        • Quality improvement (6 sessions)
        • Patient safety (2 sessions)
        • Time management
        • Physician wellness and fatigue (2 sessions)
        • Work–life balance
        • Technology resources (2 sessions)
      • Career Development (10 sessions)
        • Setting goals
        • Individual development plans (2 hours)
        • Career pathways
        • Scientific writing
        • Grants
        • Job search process (3 sessions)
        • Interviewing
        • Productivity and promotion
      • Research and Scholarship (5 sessions)
        • Writing an abstract
        • Preparing a poster
        • Giving an oral presentation
        • Scholarship workshop (1.5 hours)
        • Annual research symposium (2 hours)
      Each session is 1 hour long unless otherwise indicated.

      Outputs: Participants

      In our needs assessment, 16 (80%) of 20 subspecialty program directors stated that sharing curriculum resources was important. Furthermore, both program directors and fellows expressed interest in having a centralized educational program, especially with curricula on preparing for an academic career. The most frequently identified barriers to participation reported by both groups were time, resources, and administrative support. Today, FC serves more than 150 BCM subspecialty fellows in 18 ACGME-accredited and 2 non-ACGME-accredited (ie, academic general pediatrics, pediatric hospital medicine) pediatric training programs. Fellows and program directors from 3 other departments (eg, radiology) have joined FC since its inception. We tracked fellow participation in the program offerings as an output in our logic model. The number of attendees at the FC orientation increased from 49 fellows from 11 programs in 2011 to 75 fellows in 23 programs in 2015. The attendance at each FC session increased from 16 to 36 (average 25) fellows in 2011 to 14 to 53 (average 31) fellows in 2014–2015. Sessions were designed to meet specific needs for each year of fellowship training as directed by subspecialty program directors. Participation by all 20 pediatric subspecialty fellowship programs has been sustained.

      Outcomes

      After each session, fellows completed an evaluation of the quality of speakers, content, and relevance. On the basis of evaluation results and comments, fellows viewed learning experiences in FC as both relevant and a valuable addition to their training (eg. “covering topics not discussed in other forums”). They valued opportunities to meet with fellows in other subspecialties and to ask for guidance in their professional development. Our steering committee meets annually to review the program's content evaluations and progress and to identify areas for further expansion.
      Given fellows' restricted time and competing clinical obligations, mandatory (eg, professionalism) versus voluntary (eg, global health) curricula have been identified. As FC grew, we used process evaluation data to make decisions about sessions to retain, remove, or revise.
      FC provides an infrastructure for session logistics, planning, and communication among all participants. A centralized calendar and reminders of all FC events are distributed electronically and at monthly program director meetings. A dedicated Web site provides access to a repository of curriculum content and other resources. Program coordinators network to share expertise, ideas, and, administrative responsibilities. An unanticipated benefit is the mentoring of new coordinators, which has evolved into a separate program, entitled Coordinators College, that gives administrators needed support and opportunities for professional development.

      Next Steps

      Our next steps will be to reevaluate our long-term outcomes (impact) as envisioned and prioritize curriculum content areas that were most effective (Table). On the basis of feedback from learners and program directors, we will reorganize our content into a tiered curriculum based on the fellows' year of training. This tiered approach is intended to balance the required common curriculum with various individualized subspecialty curricula.
      FC provides an excellent opportunity for faculty development of clinician educators in our department. Several FC faculty have received teaching awards and produced scholarly products for their educational contributions in FC. Therefore, we will expand the pairing of junior faculty with seasoned clinician educators in this program so that they may gain mentored experience in educational presentations, evaluation, scholarship, innovation, and leadership. To date, our experience suggests that investment in a multidisciplinary program is a valuable addition to specialty-specific curriculum components and benefits fellows, clinician educators, program coordinators, and program directors.

      Conclusions

      Outcomes-logic models are helpful tools for guiding the development, implementation, and evaluation of a multifaceted program.
      W. K. Kellogg Foundation
      Using Logic Models to Bring Together Planning, Evaluation, and Action: Logic Model Development Guide.
      • Armstrong E.G.
      • Barsion S.J.
      Using an outcomes-logic model approach to evaluate a faculty development program for medical educators.
      • Rama J.A.
      • Campbell J.R.
      • Balmer D.F.
      • et al.
      Investing in future pediatric subspecialists: a fellowship curriculum that prepares for the transition to academic careers.
      • Starmer A.J.
      • O'Toole J.K.
      • Rosenbluth G.
      • et al.
      Development, implementation and dissemination of the I-PASS handoff curriculum: a multisite educational intervention to improve patient handoffs.
      • Parker K.
      • Burrows G.
      • Nash H.
      • Rosenblum N.D.
      Going beyond Kirkpatrick in evaluating a clinician scientist program: it's not “if it works” but “how it works.”.
      This framework, as shown in the Figure, helped us identify existing resources (inputs) and organize implementation (activities and participants). It has also been useful for evaluating if we are achieving our desired outcomes.
      Supportive departmental leadership, collaborative participants, and existing institutional resources are essential to develop an innovative curriculum that overcomes the obstacles commonly associated with having various silos of graduate medical education training. By tracking our outputs and outcomes, we have learned valuable lessons. Identifying existing educational resources or programs can help establish a foundation on which to build, and borrowing resources from those programs builds opportunities for collaboration. Planning with desired outcomes in mind provides a framework for evaluating and improving the process. We found that identifying a network of committed faculty across subspecialties and roles is a great asset for success and sustainability.

      Acknowledgments

      Supported in part by Texas Children's Hospital Educational Scholarship Project Award, 2011, 2012. The authors wish to thank Dr Mary Anne Schaffer, University of California, San Francisco, for her mentoring and encouragement, without whose creativity FC would not have been conceived. We thank Dr Mark W. Kline, chairman of the Department of Pediatrics, Baylor College of Medicine; Dr B. Lee Ligon for editorial assistance; and Cindy Gaskill.
      Presented in part at the Accreditation Council of Graduate Medical Education Annual Conference, Orlando, Fla, March 2013; the Pediatric Academic Societies annual meeting, Washington, DC, May 2013; and the Association of Pediatric Program Directors annual meeting, Washington, DC, April 2014.

      References

        • Accreditation Council for Graduate Medical Education
        ACGME program requirements for graduate medical education in the subspecalties of pediatrics.
        (Available at:) (Accessed July 1, 2013)
      1. American Board of Pediatrics. Changes in training requirements for subspecialty certification. Available at: https://www.abp.org/content/general-criteria-subspecialty-certification. Accessed August 14, 2010.

        • Federation of Pediatric Organizations
        Policy statement—Pediatric fellowship training.
        Pediatrics. 2004; 114: 295
        • Mink R.
        • Norwood V.
        • Degnon L.
        • et al.
        Council of Pediatric Subspecialties (CoPS): the first five years.
        Pediatrics. 2012; 130: 335
        • Gupta M.
        • Ringer S.
        • Tess A.
        • et al.
        Developing a quality and safety curriculum for fellows: lessons learned from a neonatology fellowship program.
        Acad Pediatr. 2014; 14: 47-53
        • Martinez J.A.
        • Koyama T.
        • Acra S.
        • et al.
        Nutrition education for pediatric gastroenterology, hepatology, and nutrition fellows: survey of NASPGHAN fellowship training programs.
        J Pediatr Gastroenterol Nutr. 2012; 55: 131-135
        • Hastings C.
        • Wechsler D.S.
        • Stine K.C.
        • et al.
        Consensus on a core curriculum in American training programs in pediatric hematology-oncology: a report from the ASPHO Training Committee.
        Pediatr Hematol Oncol. 2007; 24: 503-512
        • Collins J.
        Good to Great and the Social Sectors.
        HarperCollins, New York, NY2005
        • W. K. Kellogg Foundation
        Using Logic Models to Bring Together Planning, Evaluation, and Action: Logic Model Development Guide.
        W. K. Kellogg Foundation, Battle Creek, MI2004
        • Armstrong E.G.
        • Barsion S.J.
        Using an outcomes-logic model approach to evaluate a faculty development program for medical educators.
        Acad Med. 2006; 81: 483-488
        • Rama J.A.
        • Campbell J.R.
        • Balmer D.F.
        • et al.
        Investing in future pediatric subspecialists: a fellowship curriculum that prepares for the transition to academic careers.
        Med Educ Online. 2015; 20: 26714
        • Starmer A.J.
        • O'Toole J.K.
        • Rosenbluth G.
        • et al.
        Development, implementation and dissemination of the I-PASS handoff curriculum: a multisite educational intervention to improve patient handoffs.
        Acad Med. 2014; 89: 876-884
        • Parker K.
        • Burrows G.
        • Nash H.
        • Rosenblum N.D.
        Going beyond Kirkpatrick in evaluating a clinician scientist program: it's not “if it works” but “how it works.”.
        Acad Med. 2011; 86: 1389-1396