Advertisement

Exploring the Educational Value of Patient Feedback: A Qualitative Analysis of Pediatric Residents' Perspectives

Published:December 12, 2016DOI:https://doi.org/10.1016/j.acap.2016.10.020
      Since the introduction of the Accreditation Council for Graduate Medical Education (ACGME) Pediatric Milestones, program directors have sought to develop meaningful assessment strategies to evaluate residents' performance and identify their learning needs.
      • Hicks P.J.
      • Englander R.
      • Schumacher D.J.
      • et al.
      Pediatrics Milestone Project: next steps toward meaningful outcomes assessment.
      Of the 21 pediatric competency domains, 8 address interpersonal communication skills and professionalism. Six of these (ICS1, ICS2, PBLI4, PROF1, PROF2, PROF3) may be optimally assessed through multisource feedback, including feedback from patients and families.
      • Schwartz A.
      • Margolis M.J.
      • Multerer S.
      • et al.
      APPD LEARN–NBME Pediatrics Milestones Assessment Group. A multi-source feedback tool for measuring a subset of Pediatrics Milestones.
      While patient feedback can lead to learning and behavior change, the evidence supporting these assumptions is variable, particularly among trainees.
      • Ferguson J.
      • Wakeling J.
      • Bowie P.
      Factors influencing the effectiveness of multisource feedback in improving the professional practice of medical doctors: a systematic review.
      • Sargeant J.
      • Mann K.
      • Ferrier S.
      Exploring family physicians' reactions to multisource feedback: perceptions of credibility and usefulness.
      • Sargeant J.
      • Mann K.
      • Sinclair D.
      • et al.
      Challenges in multisource feedback: intended and unintended outcomes.
      • Reinders M.E.
      • Ryan B.L.
      • Blankenstein A.H.
      • et al.
      The effect of patient feedback on physicians' consultation skills: a systematic review.
      • Brinkman W.B.
      • Geraghty S.R.
      • Lanphear B.P.
      • et al.
      Effect of multisource feedback on resident communication skills and professionalism: a randomized controlled trial.
      • Reinders M.E.
      • Blankenstein A.H.
      • van der Horst H.E.
      • et al.
      Does patient feedback improve the consultation skills of general practice trainees? A controlled trial.
      • Evans R.G.
      • Edwards A.
      • Evans S.
      • et al.
      Assessing the practising physician using patient surveys: A systematic review of instruments and feedback methods.
      Two recent systematic reviews found no conclusive evidence for the effectiveness of patient feedback in promoting behavior change.
      • Reinders M.E.
      • Ryan B.L.
      • Blankenstein A.H.
      • et al.
      The effect of patient feedback on physicians' consultation skills: a systematic review.
      • Evans R.G.
      • Edwards A.
      • Evans S.
      • et al.
      Assessing the practising physician using patient surveys: A systematic review of instruments and feedback methods.
      One study found that pediatric residents who received patient and nurse feedback with a faculty coach had improved ratings on communicating effectively with patients from nurses, but not improved ratings from parents.
      • Brinkman W.B.
      • Geraghty S.R.
      • Lanphear B.P.
      • et al.
      Effect of multisource feedback on resident communication skills and professionalism: a randomized controlled trial.
      Others have reported that residents may value patient feedback less than feedback from faculty and peers.
      • Thomas P.A.
      • Hellmann D.B.
      Residents' attitudes toward patient feedback.
      Despite these findings, medical educators regard patient feedback as critical to resident learning and professional development, and attainment of such feedback remains an important goal of programs nationwide.
      • Sargeant J.
      • Bruce D.
      • Campbell C.
      Practicing physicians' needs for assessment and feedback as part of professional development.
      As program directors are asked to invest more time and resources to gather patient feedback, identifying what factors promote learning from this feedback will become increasingly more important. Some studies have explored factors that enhance learning from faculty, nurse and peer feedback, but they have not focused on feedback from patients and families.
      • Watling C.
      • Driessen E.
      • van der Vleuten C.P.
      • et al.
      Learning from clinical work: the roles of learning cues and credibility judgements.
      • Delva D.
      • Sargeant J.
      • Miller S.
      • et al.
      Encouraging residents to seek feedback.
      • Reddy S.T.
      • Zegarek M.H.
      • Fromme H.B.
      • et al.
      Barriers and facilitators to effective feedback: A qualitative analysis of data from multispecialty resident focus groups.
      This study is the first in the pediatric setting to identify what factors facilitate resident use of feedback from patients and families. Specifically, we sought to qualitatively explore: 1) how residents engage with patient and family feedback on their communication and interpersonal skills, and 2) the specific factors that enhance the educational value of this feedback, as identified by pediatric residents.

      Methods

      We administered a qualitative survey to all pediatric residents at Stanford School of Medicine in January 2015 after implementation of a pilot patient feedback program in July–September 2014.

      Pilot Patient Feedback Program

      From July through September, 4 trained non-MD research assistants administered the Communication Assessment Tool (CAT)
      • Makoul G.
      • Krupat E.
      • Chang C.H.
      Measuring patient views of physician communication skills: development and testing of the Communication Assessment Tool.
      by paper or electronic tablet to patients and their families to evaluate residents' communication and interpersonal skills. The CAT asks respondents to rate 14 dimensions of the physician's communication and interpersonal skills using a 5-point rating scale (poor to excellent). We modified the CAT to include 2 open-ended questions to probe for specific details regarding residents' skills: 1) What did you like about this resident's communication? and 2) How can this resident improve? The CAT has been validated across a range of subspecialties.
      • Makoul G.
      • Krupat E.
      • Chang C.H.
      Measuring patient views of physician communication skills: development and testing of the Communication Assessment Tool.
      • Ferranti D.E.
      • Makoul G.
      • Forth V.E.
      • et al.
      Assessing patient perceptions of hospitalist communication skills using the Communication Assessment Tool (CAT).
      • Myerholtz L.
      • Simons L.
      • Felix S.
      • et al.
      Using the Communication Assessment Tool in family medicine residency programs.
      • Osorio S.N.
      • Ward M.J.
      • Siew L.
      • et al.
      Assessment of pediatric residents' communication and interpersonal skills during family-centered rounds.
      We administered the CAT to a convenience sample of patients and families on 5 inpatient units. All resident-covered English- and Spanish-speaking patients were eligible and were approached during the times research assistants were available. Verbally consenting participants were shown pictures of the residents on that service and were asked to provide feedback to only the residents they could visually identify. Participants were asked to evaluate a resident only once, but they could evaluate all of the residents they worked with.
      In October 2014, 2 authors (CR and TC) compiled the CATs and created summary reports for the residents that included a mean performance rating for each CAT item, a composite mean for all items, and verbatim patient and family comments from the open-ended questions. Reports were e-mailed to the residents to review for their own learning. Twenty-eight (34%) of 82 residents received feedback and summary reports during this time period.

      Qualitative Survey Administration

      In January 2015, all 82 residents were asked to complete a written, open-ended survey assessing their communication skills and experiences receiving patient feedback, either formally through the summary reports or informally during clinical care. Questions were as follows: 1) What are your strengths in communication with patients and families? 2) How can you improve your communication with patients and families? 3) What concerns do you have about collecting and receiving feedback from patients and families? 4) What have you found helpful or unhelpful about receiving patient feedback? 5) Did you feel the feedback you received accurately represented your skills? and 6) How, if at all, has receiving this feedback changed the way you communicate with patients and families?
      Surveys were administered in person at dedicated meeting times. Each resident was assigned a unique code that was linked to surveys for completion tracking and allowed us to identify which residents received summary reports. Residents who did not complete the survey were invited via e-mail to complete it. Only responses collected within 2 weeks of initial administration were included in the analysis. Participation was voluntary, and no incentives were offered. Data were transferred to a secure electronic database before analysis.

      Data Analysis

      Data were thematically analyzed by 2 authors (AB and CR) using conventional content analysis.
      • Corbin J.
      • Strauss A.
      Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory.
      • Hsieh H.F.
      • Shannon S.E.
      Three approaches to qualitative content analysis.
      Conventional content analysis seeks to describe a phenomenon through the systematic coding and interpretation of data to identify themes.
      • Hsieh H.F.
      • Shannon S.E.
      Three approaches to qualitative content analysis.
      This approach is used to develop an understanding of phenomena not well understood and is optimally employed when theory development is not a research goal.
      • Hsieh H.F.
      • Shannon S.E.
      Three approaches to qualitative content analysis.
      We analyzed the data to explore whether there were any differences between the responses of residents who received written feedback and those who did not. This analysis occurred in several stages. First, we individually read all responses to develop a preliminary list of codes (ie, tags applied to statements or words to catalogue concepts). We assigned codes both deductively (prespecified codes from the survey questions) such as “communication strengths,” and inductively such as “feeling like a bad doctor.” Assigning codes inductively allowed us to analyze the responses from the ground up to generate de novo insights.
      • Corbin J.
      • Strauss A.
      Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory.
      Second, we discussed our list of codes to determine whether there were any differences between the residents who received feedback and those who did not. One code (“discount feedback”) only appeared in the responses of residents who received feedback; therefore, we developed and agreed on a single code list that included “discount feedback.” Third, we reread and manually recoded the responses using our single code list and categorized codes into themes. Finally, we discussed and refined our list of themes until we reached total consensus on all themes, which were then further categorized into major domains after review by all authors. Validity of the final list of themes was established through participant review, where a subset of participants from both groups was asked to review and comment on the themes' accuracy.
      • Corbin J.
      • Strauss A.
      Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory.
      The institutional review board at Stanford University reviewed this project and deemed it exempt.

      Results

      Ninety-five percent of residents (78/82) completed the survey. Our response rate was 96% for postgraduate year (PGY) 1s (27/28), 96% for PGY2s (26/27), and 93% for PGY3s (25/27). Of the 28 residents who received patient feedback summary reports, 27 (96%) completed the survey.
      Several themes emerged within 3 core domains: 1) perceptions of strengths and areas for improvement in communication, 2) responses to positive and constructive feedback, and 3) concerns about using patient and family feedback as an educational tool. We discuss each in turn.

      Perceptions of Strengths and Areas for Improvement in Communication

      Residents were able to self-reflect on several strengths and areas for improvement independent of whether they received written patient feedback. Frequently cited strengths were explaining medical conditions and treatment plans to families, actively listening, and showing empathy. Residents commented that eliciting and listening to families' concerns was one of the most helpful skills they could offer, and many felt confident in this area. Several also reflected that taking the time to engage in conversation about these concerns demonstrated their respect and concern for patients and their families.
      Areas for improvement were spending more time with patients and families, involving families in medical decision making, using less medical jargon, and assessing patient and family understanding of treatment plans. Residents frequently commented that these skills were more difficult to implement on busy clinical services but recognized that such skills were necessary for patient-centered care.

      Responses to Feedback

      Many residents valued positive patient feedback on their communication and interpersonal skills, particularly if it aligned with their self-perceptions. Positive feedback was frequently described as “validating” and “reinforcing of strengths.” Residents responded in one of 3 ways to constructive feedback: 1) feeling like a bad doctor, 2) discounting the feedback, or 3) reflecting on the feedback to improve. Residents who received written feedback were more likely than others to discount it or question its credibility if it did not align with their self-perceptions. These residents commented that patient feedback represented “a snapshot” in time and had difficulty taking it seriously. All residents were more likely to reflect on and accept patient feedback when it aligned with their self-perceptions or if they shared this feedback with a faculty coach or advisor (Table 1).
      Table 1Resident Responses to Positive and Constructive Feedback From Patients and Families
      DomainThemeRepresentative Quotations
      Responses to positive feedbackValue positive patient feedbackI think we have to value patient feedback as doctors! I definitely value it and think it has the potential to be very helpful, especially as trainees since many of us question our own skills. It's helpful when parents say you're doing well.
      Validating to hear positive feedbackIt was rewarding to see that patients appreciated my empathy and time spent with them.
      Responses to constructive feedbackFeeling like a “bad” doctorI worry about feeling inadequate or like a bad doctor. I don't want to feel like my patients are unsatisfied with their care.
      Discounting the feedback; questioning the feedback's credibility if it does not align with residents' self-perceptionsIt's hard to receive feedback if it doesn't align with your perspectives. I received negative feedback from a parent about my communication, but I think it was because the medical situation was so difficult. I didn't really take what she said seriously because of that—it wasn't about my own skills.
      Reflecting on the feedback individually or with a coach to improveMy coach and I talked about the feedback from patients, which I thought was really helpful. She really stretched me to think about the big picture and not just getting stuck on one or two situations, which I could have easily disregarded as one-offs. One parent said I seemed rushed and it's like, yes, of course I'm rushed! But talking about it with my coach helped me think about how I'm perceived, what nonverbal and verbal indicators I'm giving to make me seem rushed and how important it is to be present and try to put that feeling aside with patients.

      Concerns About Using Patient Feedback as an Educational Tool

      All residents described concerns about using patient feedback as an educational tool. Many were concerned that patients willing to provide feedback were often highly satisfied or highly dissatisfied with their care, which could result in skewed assessments. As such, residents commented that they would be more likely to accept feedback if it was gathered from a range of patients to represent multiple perspectives. Residents believed that patient and family feedback was often influenced by a patient's diagnosis, systems issues, or team decision making, and was therefore biased and not representative of the residents' skills. They feared that personally soliciting feedback from patients could violate patient confidentiality and negatively impact the patient–provider relationship and felt feedback should be gathered by a third party. Residents were more likely to trust the accuracy of feedback that came from patients they had longitudinal relationships with. Finally, feedback that was behavior specific and timely was also perceived as more valuable than vague comments or feedback provided several weeks or months after an encounter (Table 2).
      Table 2Resident Concerns About Using Patient Feedback as an Educational Tool and Suggestions to Enhance Utility
      DomainThemeRepresentative Quotation
      Concerns about using patient/family feedback as an educational toolFeedback captures highly satisfied or dissatisfied patients and therefore is not representative of residents' skillsI think feedback can be biased. Parents want to give feedback at the extremes—either really good or really bad, which may be less helpful to us.
      Feedback often relates to patient's diagnosis, systems issues or team decision-making rather than residents' skillsFeedback may be situational or based on parents'/patients' overall hospital admission or diagnosis and not based on our own skills, so it's not really useful. I tend not to pay too much attention to that feedback.
      Personally soliciting feedback from patients could negatively impact the patient-provider relationshipI worry there's a power imbalance when you ask for feedback that could affect our relationship. [Patients] may feel that giving negative feedback would negatively impact their care.
      Suggestions to enhance the usefulness of feedbackMust be gathered from multiple patients and/or families to be representative of a range of encountersThere has to be a “critical mass” of feedback so general trends and areas of improvement can be identified.
      Must be specificHelpful feedback is specific feedback! We want specific feedback, not just general comments. We want to know the specifics, the behaviors or skills we can actually change or improve on. It really benefits us to have this because many of us want to be better; we want to know how to improve.
      Must be timelyFeedback should be delivered as soon after the interaction as possible. It's easier to reflect on it when it's in context and when I remember the encounter.

      Discussion

      Implications and Recommendations

      This study is the first in the pediatric setting to offer insight into how residents perceive and engage with patient and family feedback, an important but understudied component of multisource feedback.
      • Reinders M.E.
      • Ryan B.L.
      • Blankenstein A.H.
      • et al.
      The effect of patient feedback on physicians' consultation skills: a systematic review.
      • Evans R.G.
      • Edwards A.
      • Evans S.
      • et al.
      Assessing the practising physician using patient surveys: A systematic review of instruments and feedback methods.
      • Watling C.
      • Driessen E.
      • van der Vleuten C.P.
      • et al.
      Learning from clinical work: the roles of learning cues and credibility judgements.
      • Delva D.
      • Sargeant J.
      • Miller S.
      • et al.
      Encouraging residents to seek feedback.
      • Reddy S.T.
      • Zegarek M.H.
      • Fromme H.B.
      • et al.
      Barriers and facilitators to effective feedback: A qualitative analysis of data from multispecialty resident focus groups.
      While many residents valued positive patient feedback, they were skeptical of the feedback's accuracy and described several concerns about its utility as an educational tool. These factors included the nature of the feedback (positive or constructive), the congruency of the feedback with residents' self-perceptions, patient-related factors, and feedback timeliness and specificity.
      Several studies on feedback from faculty have reported similar results, confirming the diverse influences on feedback acceptance.
      • Sargeant J.
      • Mann K.
      • Ferrier S.
      Exploring family physicians' reactions to multisource feedback: perceptions of credibility and usefulness.
      • Sargeant J.
      • Mann K.
      • Sinclair D.
      • et al.
      Challenges in multisource feedback: intended and unintended outcomes.
      • Sargeant J.
      • Bruce D.
      • Campbell C.
      Practicing physicians' needs for assessment and feedback as part of professional development.
      • Sargeant J.M.
      • Mann K.V.
      • van der Vleuten C.P.
      • et al.
      Reflection: a link between receiving and using assessment feedback.
      • DeNisi A.S.
      • Kluger A.N.
      Feedback effectiveness: can 360-degree appraisals be improved?.
      This study demonstrates that feedback from patients and families may introduce additional and unique challenges that further affect feedback acceptance and use. Our findings suggest that patient and family feedback may need to be gathered by a third party, include data from a diverse range of patients and families, and include perspectives of patients and families with whom residents have worked longitudinally in order to be valued and accepted. To address this, programs may benefit from investing in nontrainee feedback gatherers who focus their feedback-collecting efforts in longitudinal inpatient settings or continuity clinic.
      Optimizing the value residents place on patient and family feedback also requires attention to the way it is delivered. One notable finding was that residents were more likely to discount patient and family feedback if it did not align with their self-perceptions. Genuine learning from feedback involves giving it serious consideration, reflecting on how it pertains to one's practice, interpreting it, and considering how to accept and respond to it.
      • Sargeant J.M.
      • Mann K.V.
      • van der Vleuten C.P.
      • et al.
      Reflection: a link between receiving and using assessment feedback.
      Feedback that is consistent with residents' self-perceptions may be easier to cognitively and emotionally engage with because it highlights residents' strengths, builds their confidence, and reinforces known skill deficits. Feedback that counters residents' self-perceptions may elicit emotional responses such as surprise, confusion, anger, denial, sadness, or disappointment, and may lead to demotivation and poorer performance rather than improvement.
      • Sargeant J.M.
      • Mann K.V.
      • van der Vleuten C.P.
      • et al.
      Reflection: a link between receiving and using assessment feedback.
      While this feedback may be difficult to engage with, we believe it may be even more informative and capable of transforming clinical practice, particularly if it comes from patients and families. When critically reflected on, experiences that counter one's beliefs can build fresh perspectives, prompt skill development, and motivate change.
      • DeNisi A.S.
      • Kluger A.N.
      Feedback effectiveness: can 360-degree appraisals be improved?.
      Residents in our study were more likely to reflect on and accept patient feedback when they discussed this feedback with a faculty coach or advisor. We believe patient and family feedback may be optimally delivered by a coach, advisor, or other trusted source who can facilitate the reflective processes one needs to learn from feedback and integrate it into practice. This recommendation is consistent with others who have found facilitated reflection on multisource feedback to be essential for improving its acceptability and use.
      • Reinders M.E.
      • Blankenstein A.H.
      • van der Horst H.E.
      • et al.
      Does patient feedback improve the consultation skills of general practice trainees? A controlled trial.
      • Overeem K.
      • Wollersheimh H.C.
      • Arah O.A.
      • et al.
      Factors predicting doctors' reporting of performance change in response to multisource feedback.
      • Overeem K.
      • Lombarts M.J.
      • Arah O.A.
      • et al.
      Three methods of multi-source feedback compared: a plea for narrative comments and coworkers' perspectives.
      • Owens J.
      Is multi-source feedback (MSF) seen as a useful educational tool in primary care? A qualitative study.
      Coaches or advisors may be able to ease emotional distress caused by constructive feedback, provide a third-party perspective on the resident's communication skills, help residents identify realistic and achievable goals from the range of feedback received, and provide ongoing support and follow-up to residents on their goal setting.
      Patient and family feedback will invariably be subjective and situated within the context of the patient's illness and the overall care provided by the health care team. As systems for gathering patient feedback are developed and refined, residents need assistance and support from faculty to identify learning opportunities from all forms of patient and family feedback.

      Limitations

      This study has several limitations. It was conducted at a single institution, albeit with a high survey response rate. The survey was part of a pilot study on patient–provider communication in which some, but not all, residents received written patient feedback. Furthermore, this study solicited resident perspectives about their communication skills and beliefs about patient feedback and did not study actual behavior change. We do not know whether receipt of feedback resulted in behavior change or improvements in patient care, a critical area for future study.

      Conclusions

      This study highlights several strategies that can be used with trainees to promote learning from their patients and families, especially in areas of interpersonal communication and professionalism. Our findings suggest that residency programs must consider both the process by which feedback is gathered as well as how it is delivered to residents. In particular, facilitated sessions with a coach, advisor, or other trusted source to discuss patient feedback in the context of residents' self-identified strengths and weaknesses may be necessary for residents to meaningfully engage with and learn from their patients' perspectives, particularly if these perspectives do not align with residents' self-assessments. Discussions with residents about patient feedback may encourage openness to constructive comments, facilitate goal setting to improve patient care, and, most importantly, support their professional development.

      References

        • Hicks P.J.
        • Englander R.
        • Schumacher D.J.
        • et al.
        Pediatrics Milestone Project: next steps toward meaningful outcomes assessment.
        J Grad Med Educ. 2010; 2: 577-584
        • Schwartz A.
        • Margolis M.J.
        • Multerer S.
        • et al.
        APPD LEARN–NBME Pediatrics Milestones Assessment Group. A multi-source feedback tool for measuring a subset of Pediatrics Milestones.
        Med Teach. 2016; 38: 995-1002
        • Ferguson J.
        • Wakeling J.
        • Bowie P.
        Factors influencing the effectiveness of multisource feedback in improving the professional practice of medical doctors: a systematic review.
        BMC Med Educ. 2014; 14: 76
        • Sargeant J.
        • Mann K.
        • Ferrier S.
        Exploring family physicians' reactions to multisource feedback: perceptions of credibility and usefulness.
        Med Educ. 2005; 39: 497-504
        • Sargeant J.
        • Mann K.
        • Sinclair D.
        • et al.
        Challenges in multisource feedback: intended and unintended outcomes.
        Med Educ. 2007; 41: 583-591
        • Reinders M.E.
        • Ryan B.L.
        • Blankenstein A.H.
        • et al.
        The effect of patient feedback on physicians' consultation skills: a systematic review.
        Acad Med. 2011; 86: 1426-1436
        • Brinkman W.B.
        • Geraghty S.R.
        • Lanphear B.P.
        • et al.
        Effect of multisource feedback on resident communication skills and professionalism: a randomized controlled trial.
        Arch Pediatr Adolesc Med. 2007; 161: 44-49
        • Reinders M.E.
        • Blankenstein A.H.
        • van der Horst H.E.
        • et al.
        Does patient feedback improve the consultation skills of general practice trainees? A controlled trial.
        Med Educ. 2010; 44: 156-164
        • Evans R.G.
        • Edwards A.
        • Evans S.
        • et al.
        Assessing the practising physician using patient surveys: A systematic review of instruments and feedback methods.
        Fam Pract. 2007; 24: 117-127
        • Thomas P.A.
        • Hellmann D.B.
        Residents' attitudes toward patient feedback.
        Acad Med. 1999; 74: 90-91
        • Sargeant J.
        • Bruce D.
        • Campbell C.
        Practicing physicians' needs for assessment and feedback as part of professional development.
        J Contin Educ Health Prof. 2013; 33: S54-S62
        • Watling C.
        • Driessen E.
        • van der Vleuten C.P.
        • et al.
        Learning from clinical work: the roles of learning cues and credibility judgements.
        Med Educ. 2012; 46: 192-200
        • Delva D.
        • Sargeant J.
        • Miller S.
        • et al.
        Encouraging residents to seek feedback.
        Med Teach. 2013; 35: e1625-e1631
        • Reddy S.T.
        • Zegarek M.H.
        • Fromme H.B.
        • et al.
        Barriers and facilitators to effective feedback: A qualitative analysis of data from multispecialty resident focus groups.
        J Grad Med Educ. 2015; 7: 214-219
        • Makoul G.
        • Krupat E.
        • Chang C.H.
        Measuring patient views of physician communication skills: development and testing of the Communication Assessment Tool.
        Patient Educ Counsel. 2007; 67: 333-342
        • Ferranti D.E.
        • Makoul G.
        • Forth V.E.
        • et al.
        Assessing patient perceptions of hospitalist communication skills using the Communication Assessment Tool (CAT).
        J Hosp Med. 2010; 5: 522-527
        • Myerholtz L.
        • Simons L.
        • Felix S.
        • et al.
        Using the Communication Assessment Tool in family medicine residency programs.
        Family Med. 2010; 42: 567-573
        • Osorio S.N.
        • Ward M.J.
        • Siew L.
        • et al.
        Assessment of pediatric residents' communication and interpersonal skills during family-centered rounds.
        Hosp Pediatr. 2012; 2: 85-92
        • Corbin J.
        • Strauss A.
        Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory.
        Sage, Thousand Oaks, Calif2015
        • Hsieh H.F.
        • Shannon S.E.
        Three approaches to qualitative content analysis.
        Qual Health Res. 2005; 15: 1277-1288
        • Sargeant J.M.
        • Mann K.V.
        • van der Vleuten C.P.
        • et al.
        Reflection: a link between receiving and using assessment feedback.
        Adv Health Sci Educ Theory Pract. 2009; 14: 399-410
        • DeNisi A.S.
        • Kluger A.N.
        Feedback effectiveness: can 360-degree appraisals be improved?.
        Acad Manag Exec. 2000; 14: 129-139
        • Overeem K.
        • Wollersheimh H.C.
        • Arah O.A.
        • et al.
        Factors predicting doctors' reporting of performance change in response to multisource feedback.
        BMC Med Educ. 2012; 12: 52
        • Overeem K.
        • Lombarts M.J.
        • Arah O.A.
        • et al.
        Three methods of multi-source feedback compared: a plea for narrative comments and coworkers' perspectives.
        Med Teach. 2010; 32: 141-147
        • Owens J.
        Is multi-source feedback (MSF) seen as a useful educational tool in primary care? A qualitative study.
        Educ Prim Care. 2010; 21: 180-185