If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Address correspondence to Heather B. Howell, MD, Department of Pediatrics, New York University Grossman School of Medicine, 317 East 34th Street, Suite 902, New York, NY 10016
Department of Pediatrics, New York University Grossman School of Medicine (HB Howell, PV Desai, L Vrablik, and FH Levy), New York, NYSala Institute for Child and Family Centered Care, Hassenfeld Children's Hospital (M McGrath, R Ramsey and FH Levy), New York, NY
Curricula designed to teach and assess the communication skills of pediatric residents variably integrates the parent perspective. We compared pediatric residents' communication skills in an objective structured clinical exam (OSCE) case as assessed by Family Faculty (FF), parents of pediatric patients, versus standardized patients (SP).
Methods
Residents participated in an OSCE case with a SP acting as a patient's parent. We compared resident performance as assessed by FF and SP with a behaviorally-anchored checklist. Items were rated as not done, partly done or well done, with well-done indicating mastery. The residents evaluated the experience.
Results
42 residents consented to study participation. FF assessed a lower percentage of residents as demonstrating skill mastery as compared to SP in 19 of the 23 behaviors. There was a significant difference between FF and SP for Total Mastery Score and Mastery of the Competency Scores in three domains (Respect and Value, Information Sharing and Participation in Care and Decision Making). The majority of residents evaluated the experience favorably.
Conclusion
Involving parents of pediatric patients in the instructive and assessment components of a communication curriculum for pediatric residents adds a unique perspective and integrates the true stakeholders in parent—physician communication.
Successful parent-physician communication is foundational in pediatrics. Parents of pediatric patients provide a different perspective than standardized patients when assessing pediatric resident communication skills in an OSCE case. Including parents in communication skill training for pediatric residents integrates true stakeholders.
Clear physician-parent communication that promotes partnership is one of the foundations of practicing effective pediatric medicine. A physician's ability to successfully communicate has a strong impact on patient experience and satisfaction.
Given the importance of communication to health outcomes, it is not surprising that the Accreditation Council for Graduate Medical Education (ACGME) emphasizes the need to develop this crucial skill.
Despite the commonplace presence of communication curricula in pediatric residency programs, there is inconsistency and variability in how programs teach these skills to their trainees and assess competency.
Simulation-based education, specifically an Objective Structured Clinical Examination (OSCE), has well documented benefits and can fill the gap in difficult to assess competencies such as interpersonal communication.
Assessing advanced communication skills via objective structured clinical examination: a comparison of faculty versus self, peer, and standardized patient assessors.
In pediatrics, standardized patients (SPs) can act in the role of a patient's parent and provide learner assessments, but the degree to which this reflects the true parent perspective is unknown.
Including the parent perspective is especially important to advancing a culture of safe, patient and family-centered care.
We include parent perspective in the communication skills training for our pediatric residents through a partnership with the Sala Patient and Family Faculty Program. Family Faculty (FF) are parents of pediatric patients cared for at our institution who employ their personal experiences in the observation, evaluation, and provision of feedback to trainees and staff. We developed an OSCE case for pediatric residents focused on physician-parent communication that utilized FF as educators and assessors. The objective of this study is to compare pediatric resident communication skills in an OSCE case as assessed by FF versus SPs.
Methods
Study Design
This is a cohort study of pediatric residents in post graduate years (PGY) 1-3 at New York University Grossman School of Medicine. Residents participated in a single OSCE case as the initial session of a new communication curriculum for our residency program. The participants did not have any previous communication skills training during their residency. Participation in the OSCE case is an educational requirement for all residents, but data for research purposes were collected only from those residents who consented to study participation as part of our institutional review board-approved Medical Education Research Registry.
Training materials for the OSCE case were developed by conducting in-person focus groups and mock-simulations with FF, SPs, and clinicians at New York Simulation Center. There were 8 participating SPs who are professional actors and who are paid $25 per hour. There were 10 FF who volunteer their time. The SPs and FF underwent approximately 6 hours of training that was led by family engagement experts from the Sala Institute for Child and Family Centered Care at Hassenfeld Children's Hospital. They were taught the core competencies of patient and family-centered care
and the key behaviors that the residents would be evaluated on. They were trained how to objectively complete the assessment tool using specific behavioral anchors and how to give feedback to the resident aligned with the tool and the case specific learning objectives. The SPs received additional coaching from FF to ensure authenticity and standardization in their character portrayal and reactions.
The OSCE Case
The residents individually participated in a single scenario OSCE in which they were tasked with disclosing a medical error to the SP, who was acting in the role of the patient's parent (Supplement A). The OSCE case was designed by members of our departmental education team, reviewed by FF with integration of their feedback and pilot tested on faculty. The scenario required the resident to disclose that an incorrect lab test had been ordered and that the patient would require repeat venipuncture. The residents were told that the learning objective was to increase comfort and skill in having difficult conversations and that they were to focus on communication rather than the specific error. This particular case required the resident to disclose the error, respond to the parent's emotion and then partner with the parent in an actionable next step (ie, a repeat blood draw). The 10-minute encounter between the resident and SP was observed by a FF, a pediatric clinical faculty member and trained debrief facilitator through one-way glass. The debrief facilitators have clinical backgrounds in social work, psychology, psychiatry or child life, and facilitate groups as part of their professional role. They undergo annual training specifically about debriefing with FF and they use a facilitator guide to ensure consistency in the debrief sessions. Prior to the case starting, the resident was oriented to the scenario, informed of the observers and reminded that simulation is a safe learning environment. Following the encounter, there was a 20-minute, 360 degree debrief led by the trained facilitator that included the resident, the SP, the FF and the clinical faculty member.
Assessment
After the conclusion of the live scenario, but prior to the debrief, the FF and SP each independently completed a 23 item behaviorally anchored assessment tool, the Patient and Family Centered Communication Assessment Tool (Supplement B). The tool was adapted from the NYU Communication Skills Assessment Tool (Supplement C), which is a behaviorally anchored checklist with previously published evidence of reliability and validity.
With input from clinicians and FF, the language of the tool was modified to be family centered and the items were grouped into 5 domains (Respect and Value, Information Sharing, Participation in Care and Decision-Making, Follow-up and Next Steps, and Working as a Team) that reflect and expand upon the core concepts of patient and family centered care from the Institute for Patient and Family Centered Care.
The tool was refined through an iterative process in which clinical faculty, FF, and SPs participated in pilot OSCEs and provided feedback on the tool. Finally, a pilot OSCE was done with pediatric subspecialty fellows showing feasibility of the tool to assess trainees. In alignment with the NYU Communication Skills Assessment Tool, the items were rated as not done, partly done or well done, with well-done indicating mastery. After the debrief, the residents each completed a three question evaluation of the curriculum using a 5-point Likert scale in which they rated statements from very untrue to very true.
Statistical Analysis
For each behavior, the percent of residents who achieved “well-done” was calculated for the FF and SP assessments. For the 4 domains with more than one item, a Mastery of the Competency Score was determined by collapsing the behaviors in that domain and calculating the average percent “well-done.” Total Mastery Score was determined by collapsing all 23 behaviors and calculating the average percent “well-done.” A Cronbach's alpha was computed to assess internal consistency of the scales. Paired t tests were done for the Mastery of the Competency Scores and the Total Mastery Score. A nonparametric Wilcoxon signed rank test was used to examine the difference between the FF and SP ratings for the single item in the Working as a Team domain.
Results
Fifty-two (90%) of the 58 pediatric residents completed the OSCE and 42 (81%) consented to study participation. Of the participating residents 11 (26%) were PGY1, 15 (36%) were PGY2, and 16 (38%) were PGY3. Twenty-nine (69%) of the residents identify as female. A Cronbach's alpha was computed to assess internal consistency, with scores of 0.78 for the FF ratings and 0.86 for the SP ratings, indicating internal consistency for both the FF and SP score. In 19 of the 23 behaviors measured, the FF assessed a lower percentage of residents as demonstrating skill mastery as compared to the SP (Table).
TableComparison of Percent of Residents Who Demonstrated Mastery of a Behavior as Assessed by the Family Faculty (FF) Versus the Standardized Patient (SP), N = 42
Competency
Behavior
FF Assessment, n (%)
SP Assessment, n (%)
P Value
Respect and value
Introduction
29 (69)
35 (83)
Elicited patient story using appropriate questions
14 (33)
28 (67)
Acknowledged emotions and feelings
18 (43)
23 (55)
Demonstrated acceptance and lack of judgment
30 (71)
39 (93)
Affirmed that the family is a key member of the care team
13 (31)
22 (52)
Mastery of Competency Score, %
50
70
<.05
Information sharing
Nonverbal behavior
23 (55)
28 (67)
Encouraged family to ask questions, interrupt if any misinformation
17 (40)
17 (40)
Used reflective listening to confirm understanding
13 (31)
24 (57)
Avoided interruption
31 (74)
29 (69)
Used of appropriate and understandable language, avoiding jargon
22 (52)
32 (76)
Communicated concern or intention to help
20 (48)
36 (86)
Inquired about existing knowledge, ideas and readiness
10 (24)
9 (21)
Provided clear explanations/information
19 (45)
29 (69)
Honesty/Transparency
27 (64)
38 (90)
Managed of the narrative flow
18 (43)
31 (74)
Appropriately Paced of encounter
23 (55)
35 (83)
Mastery of Competency Score, %
48
67
<.05
Participation in care and decision-making
Collaborated with parent to identify and decide on plan
20 (48)
30 (71)
Elicited parent's thoughts on symptoms, underlying concerns, etc.
14 (33)
29 (69)
Mastery of Competency Score, %
40
70
<.05
Follow-up and next steps
Clarified information to ensure understanding
22 (52)
20 (48)
Provided resources
14 (33)
26 (62)
Provided follow-up plan
22 (52)
31 (74)
Closed the encounter
21 (50)
23 (55)
Mastery of Competency Score, %
47
60
.081
Working as a team
Supported fellow clinicians’ expertise
14 (33)
22 (52)
.077
Total Mastery Score, %
47
66
<.05
Table shows the number and percent of residents who scored "well-done," with "well-done" indicating mastery, for each behavior. For domains with more than one behavior, a Mastery of Competency Score was calculated by averaging the percent "well-done" for all the behaviors in that competency. The Total Mastery Score was calculated by averaging the percent "well-done" for all 23 behavior items.
On the paired t test, there was a significant difference between FF and SP on the Total Mastery Score, (mean score FF = 0.47, SP = 0.66, P = .0001). Mastery of the Competency Score in each domain was lower as assessed by the FF when compared to the SP with significant differences noted in three of five domains (Table). The paired t test for the domain Follow-up and Next Steps did not reach statistical significance, nor did the domain Working as a Team using the Wilcoxon (Figure). Resident feedback showed that 83% felt that the SP interaction was helpful to their future practice, 88% felt that the feedback they received directly from FF was helpful to their future practice, and 90% thought that the 360 degree debrief was a helpful learning tool.
FigureComparison of Total Mastery Score and Mastery of the Competency Scores as assessed by the Family Faculty (FF) versus the Standardized Patient (SP), N = 42. The figure compares the Total Mastery Score and the Mastery of Competency Scores with standard deviation. The Total Mastery Score was calculated by collapsing all 23 behaviors and calculating the percent “well done” for all items. For the 4 domains with more than one item, the Mastery of the Competency Score was calculated by averaging the percent "well-done" for all the behaviors within that domain.
In this study, we found that FF, parents of pediatric patients, assessed pediatric residents differently than SPs in a communication skills OSCE case scoring them lower in all domains. Previous studies show that including the parent perspective in communication skills training is likely valuable, but the ideal approach is unclear.
The real clinical environment can be a high-stakes setting for residents to hone complex communication skills. There is also inconsistency in the ability to standardly assess residents in real encounters since the content of the exchange and the aptitude of the parent to provide effective feedback can vary widely. Studies also show that while residents value patient and family feedback from real clinical encounters, they are skeptical about accuracy and have concern about the educational utility.
Many residency programs use simulation-based education, such as an OSCE with SPs, to standardize content delivery and assessment of communication skills, but the degree to which the assessments done by SPs reflect the perspective of parents of real pediatric patients is unknown. The benefits of simulation-based education and assessment, specifically an OSCE, are well documented.
Assessing advanced communication skills via objective structured clinical examination: a comparison of faculty versus self, peer, and standardized patient assessors.
It is uniquely valuable for learners to refine skills without impact to real patients, especially with a complex and vital skill such as communication. An OSCE targeting communication allows experiential learning and serves as an educational intervention while also providing skill assessment. SPs are trained to portray a medical scenario in a consistent way and can reliably perform checklist-based assessment.
Assessing advanced communication skills via objective structured clinical examination: a comparison of faculty versus self, peer, and standardized patient assessors.
Previous studies of resident communication skills comparing SP versus adult patient assessments found that SPs were more stringent in their scoring, while our study found FF were less likely than SPs to assess residents as having met skill mastery.
In this study, the FF and SPs who participated in the OSCE case underwent similar training prior to the OSCE case. Despite consistent training and the use of a behaviorally anchored assessment tool that defined what was considered skill mastery, the differences in assessments were significant in three domains. This leads us to infer that the perspective of the assessor plays a role in how resident behavior is interpreted. There are various factors that possibly contribute to the differences seen. While FF may add authenticity that better reflects the reality that residents will face in the clinical environment, their personal experience of caring for an ill child may make them less objective. On the other hand, the SP has the unique vantage of being an active participant in the scenario, rather than a passive observer, which also adds authenticity and potentially contributes to the observed differences. We do not know personal information about the SPs, such as if they are parents themselves, potentially with children who are frequent utilizers of the health care system, which is a study limitation.
This study supports the notion that multisource feedback is likely valuable in communication skills training. Interestingly, the three competencies that we found to be significantly different (Respect and Value, Information Sharing and Participation in Care and Decision Making) are rooted in a resident's ability to convey empathy and form therapeutic partnerships. We found that the most disparate behaviors within the three significant domains were a resident's ability to elicit a parent's thoughts on their child's symptoms or ask for the parent's input (FF assessed 33% of residents to meet skill mastery vs 69% as assessed by the SP) and the ability of a resident to communicate concern or intention to help (FF assessed 48% of residents to meet skill mastery vs 85% as assessed by the SP). These findings highlight the need to include empathy skill development in communication curriculum.
This study has informed the development of our residency program communications curriculum. We include the perspective of FF in both the instructive and assessment aspects. The didactics are designed to teach the core concepts of family-centered care, to provide tools for building authentic parent-physician partnerships, and to continually expose residents to the parent perspective as a way to develop empathy. Every OSCE case in our curriculum utilizes multisource feedback from SPs, FF, and clinical faculty. The overwhelming majority of residents felt there was value in having feedback from multiple observers and that the perspective of the FF would be helpful to their future practice.
More work needs to be done to see if including FF feedback when teaching residents communication skills ultimately impacts residents’ competency. This study was limited by small sample size and the single institution participation. Lastly, we acknowledge that FF programs and the use of an OSCE as an educational tool require resources that may not be readily available to all training programs. Many hospitals have Patient and Family Advisory Committees with volunteers who could potentially be trained to provide feedback to residents on their communication skills.
Conclusion
In conclusion, we demonstrated that FF assess pediatric residents differently than SPs in a communication skills OSCE case. FF provide a unique perspective that likely adds value to resident education by integrating the true stakeholders in parent-physician communication. More work needs to be done to fully understand the root of these differences and how best to integrate multisource feedback into communication skills training.
Acknowledgments
Financial statement: No funding support.
Authorship statement: All authors have made substantial contributions to all of the following: 1) the conception and design of the study, or acquisition of data, or analysis and interpretation of data, 2) drafting the article or revising it critically for important intellectual content, and 3) final approval of the version to be submitted.
Sala Institute for Child and Family Centered Care at Hassenfeld Children's Hospital New York Simulation Center (NYSIM) a public-private partnership between NYU Langone Health and the City University of New York.
OSCE Case Authors: Cynthia J. Osman, MD, Linda R. Tewksbury, MD, Nicole L. Gerber, MD, Laura S. Schroeder. MD.
Assessing advanced communication skills via objective structured clinical examination: a comparison of faculty versus self, peer, and standardized patient assessors.
00318-1&id=gr1.jpg){kind=link}