Objectives: Residency training programs are required to assess and document residents’ competency in communication. Multisource evaluations that include direct resident observations are preferred. Previous studies have often used methods of asking faculty members to evaluate residents on behaviors they did not witness.
Methods: Twenty first year pediatric residents (interns) were evaluated preand post-FCR introduction by a faculty member, a parent, and a nurse after the encounter with the patients and their families during rounds. All participants used the CAT. Mean CAT scores and associations between the evaluator groups were determined.
Results: Nurses and faculty were significantly more likely to observe how interns communicated with the parents in the post-FCR period (P < .001). Pre-FCR, compared to parent scores, intern scores were significantly lower (P < .05) and the scores were not correlated (P = .84). Post-FCR, the difference in internparent scores disappeared and their scores were significantly correlated (r18 = 0.73, P < .001). Intern scores did not correlate with faculty and nurse scores. Cronbach a coefficients for the CAT ranged from 0.90 to 0.99.
Conclusions: Family-centered rounds provide more opportunities for direct observation of residents’ competence in accordance with Accreditation Council for Graduate Medical Education guidelines. This rounds format also encourages a climate that improves residents’ ability to self-assess. Residents’ communication skills can be assessed by using the Communication Assessment Tool in an efficient and timely manner, because all evaluators are present at the bedside during family-centered rounds.
Communication is a core skill for the practice of medicine. Effective communication strengthens patient-physician relationships, improves patient health status, recall, treatment adherence, and satisfaction.1 The Accreditation Council for Graduate Medical Education (ACGME) Outcome Project, whose purpose is to demonstrate the effectiveness and accountability of educational programs, requires that residents must demonstrate competency in communication and interpersonal skills.2
Training programs are required to assess and document residents’ competency in communication. Accurate assessment of competence requires direct observation of residents while they are performing the tasks of future real-life practice. Assessments should also be objective and come from multiple sources, including self-assessment, peers, other members of the medical team, and patients. The inclusion of multiple sources is known as multisource or 360-degree evaluation.3,4 This type of assessment provides more comprehensive accounts of resident performance.5 The multisource evaluation has been cited as potentially beneficial in helping residents to self-assess interpersonal and communication skills.6 However, previous studies on the evaluation of residents’ communication and interpersonal skills have revealed that teaching attending physicians frequently were asked to evaluate residents on behaviors that they had not observed.5,7,8
In the inpatient setting, during traditional rounds, interactions between faculty evaluators and residents are not ideally conducive to observation of residents’ humanistic skills.9 During traditional rounds, the medical team discusses a case in the classroom or the hallway, formulates a plan of care, and then enters the patient's room to deliver the plan to the patient and the family. With traditional rounds, the nurse is generally informed of the day’s plan after rounds are completed. In contrast, during family-centered rounds (FCRs) families are invited to participate in clinical decisions along with a medical team comprising a teaching attending physician, a nurse, residents, medical students, and other support staff.10,–,13 A review of the literature reveals that patients and their families prefer bedside case presentations, including teaching at the bedside.14,–,16 In addition, several authors documented that residents and physicians felt that bedside physician’s rounds provided more opportunities to model and directly observe communication skills.17,–,20 However, very little is known about the impact of FCR on medical education and assessment.
We hypothesized that FCR would add opportunities to assess residents’ communication and interpersonal skills through direct observation. The primary objective of the current study was to assess the utility of FCR for evaluation of residents’ communication skills by using the Communication Assessment Tool (CAT). The CAT is a reliable and valid instrument that measures the patient’s perception of a physician’s communication and interpersonal skills in both the outpatient and the inpatient setting.21,22 The secondary objective was to assess the reliability of the CAT as a tool for multisource evaluation of resident communication skills.
This is a secondary analysis of data from evaluations of communication and interpersonal skills in pediatric residents at an urban teaching hospital. Evaluations were available from 2 time periods, before and after introduction of FCR (herein labeled pre-FCR and post-FCR periods). Evaluations were multisource, that is, they were performed by a parent, a nurse, a faculty member, and a resident him/herself after an encounter with a parent during rounds. These evaluations were collected in compliance with ACGME requirements to document resident competencies in interpersonal and communication skills.
Our training program introduced the CAT as a 360-degree assessment tool for the evaluation of residents’ communication and interpersonal skills in June 2008. FCRs were introduced 8 months later, in January 2009. This timing provided an opportunity to study the efficacy of FCR for evaluating residents’ interpersonal and communication skills.
Twenty postgraduate year 1 residents (interns) at a large academic medical center were assessed from June 2008 through July 2009, during their inpatient rotations on the General Pediatric Floor. Interns’ communication and interpersonal skills were studied because they have the most active role communicating with the families in our model of FCR. General pediatric patients are followed up primarily by a hospitalist group. The hospitalist group comprises 5 faculty physicians who practice primarily in an inpatient setting. Admissions by private pediatricians are infrequent. Private pediatricians do not conduct FCRs in our hospital.
During their first year of training, interns have 3 to 4 inpatient pediatric rotations. On average, they carry 3 patients on the hospitalist service, and patients on other subspecialty services, as well. The hospitalist team consists of 2 interns, 1 to 2 residents, 1 to 2 medical students, and 1 faculty physician (attending). The faculty physician covers all patients on the hospitalist team. Medical students follow some of the cases on the hospitalist team, and, as such, they present those patients on rounds. However, on a day when an intern evaluation is scheduled, the intern presents the case. Five faculty physicians, 15 nurses, and 84 parents participated in the evaluation of resident communication and interpersonal skills.
The CAT was developed by Makoul and colleagues.21 This is a 15-item instrument written at the fourth grade reading level that uses a 5-point response scale with a maximum score of 75. Fourteen items focus on physician-patient communication (eg, “Talked in terms I could understand," “Treated me with respect”), and 1 focuses on staff communication. The CAT can be used for medical students and residents by eliminating the item regarding the physician’s staff and by using different introductions. We have adapted the CAT for use by other team members with simple rephrasing (eg, “Talked in terms parent could understand,” “Treated the parent with respect”).21 With the primary author’s permission, we replaced the item that focused on the staff, “The doctor’s staff treated me with respect,” with the following question: “Checked if parent agreed with the plan of care.” Parents evaluated only 1 intern each, whereas nurses and faculty evaluated on average 3 interns. For this study, the CAT was available in English, Spanish, Chinese, Korean, and Russian.
The CAT has been used successfully to assess adult hospitalists’ communication and interpersonal skills.22 In our study, all 4 evaluators used the CAT for assessment of interns’ communication skills during FCR. The reliability of the measurements was assessed with internal consistency coefficients. The validity of the appraisals of interns’ communication skills was inferred by using correlations among raters’ scores.23
Data Collection Methods
On the designated day during the second or third week of each intern’s inpatient rotation, the chief resident distributed CAT forms to parents, faculty, nurses, and the intern after encounters with designated families during rounds. The forms were coded so that names of interns and families were not recorded. The goal was to obtain at least 1 set of evaluations per intern per rotation.
A convenience sampling was used to select parents to participate. All parents of patients whose interns were evaluated on a given day were asked to participate. Parents were informed that the evaluation was confidential and gave verbal consent. Parents and nurses were informed that the purpose of using the CAT was to help to identify strengths and weaknesses in residents’ interpersonal and communication skills and that the evaluation was performed in agreement with ACGME requirements.
Forms were completed by FCR participants and collected by the chief resident on the day of the encounter with the family and were placed in each resident’s education file.
The key to link the code to intern identity was maintained in the Pediatric Residency Program Director’s office. For this study, CAT ratings were obtained from the education files without identifiers. The institutional review board at Weill Cornell Medical College approved this study as a part of a standard educational project.
All residents, nurses, and physicians were oriented to FCR before its introduction in January 2009 through a modified FCR workshop developed at Cincinnati Children’s Medical Center.24 A brief overview of the CAT was included in the workshop, but no other formal training was provided.
All available, de-identified CAT forms from the resident educational files were analyzed for completeness by generating frequency tables of individual items. Individual residents had 1 to 4 sets of evaluations from each time period. Each set contained 1 parent, 1 faculty, 1 nurse, and 1 intern self-assessment form. Multiple assessments of a resident were composited within each phase by using means.
Data from the CAT were not normally distributed (they clustered at the high end of the scale). This finding has been described in other patient-reported ratings of their experience with physicians.25 Jones and Sasser26 have suggested that any response below the maximum may indicate that some aspect of the participant’s experience was not acceptable. To overcome this limitation, it has been suggested that the percentage of “excellent” ratings per item answers, which leads to broader spectrum of scores, be calculated.21 This approach requires 20 to 30 completed CAT forms per intern. Unfortunately, because of the limited number of evaluations in our study, we used statistical corrections to normalize data for analysis. Specifically, the CAT scores were submitted to natural log transformations before analyses.
To analyze intraindividual stability, CAT scores were compared between the pre-FCR and post-FCR periods for the 15 interns who had self-evaluations and parental evaluations available from both time periods. Paired t tests were used to compare the normalized CAT scores, and Pearson correlation coefficients were used to assess associations between CAT scores. Cronbach α was used to examine the internal consistency for CAT scores.
Of 20 interns, 17 interns had their communication and interpersonal skills assessed during the first time period (pre-FCR, traditional rounds), 18 interns were assessed during the second time period (post-FCR), and 15 interns had assessments from both time periods.
A total of 108 sets (432 individual forms) were distributed. Four sets of completed forms were excluded from analysis for the pre-FCR period and 2 sets for the post-FCR period because there were errors in recording study ID number, and the forms could not be associated with any intern. Twenty-four sets had missing forms. The overall response rate was 78% (84/108). Response rates were equivalent for the 4 groups of respondents: 78% for parents (84/108), 80% for nurses, and 80% residents (86/108 for each group), and 74% for faculty (80/108). The most common reason that forms were not completed was that patients were discharged before observations could be made or parent surveys were collected.
Communication Assessment Tool: Interrater Reliability Testing
Internal consistency (Cronbach α coefficient) was calculated for all evaluator groups. Overall scale reliability was high (α-coefficients 0.9–0.99). Also, α-coefficients were high when reliability was assessed per evaluator (Table 1),
Opportunities for Direct Observation
The submitted CAT forms were reviewed for data analyses by generating frequency tables of individual items. One scoring option on the CAT is to choose “not observed,” indicating inadequate opportunity to make a judgment on an item; the other 5 options indicate frequencies with which a given behavior was observed. This review showed that nurses had chosen “not observed” on all CAT items for 32 of 44 forms (73%) and faculty had chosen “not observed” on all CAT items for 38 of 44 forms (86%). Forms marked with all items “not observed” could not be further analyzed.
Nurses and faculty were significantly more likely to provide responses indicating opportunity to observe intern behaviors on the CAT forms in the post-FCR period than in the pre-FCR period. Nurses indicated adequate observation for 29 of 34 post-FCR forms (85%; Χ21 = 23.6, P < .001), whereas they had made the same judgment in only 27% in the pre-FCR phase. Faculty indicated adequate observation for 32 of 34 post-FCR forms (94%; Χ21 = 18.7, P < .001), compared with 14% in the pre-FCR phase (Fig 1).
Parents and interns had high rates of indicating adequate observation from both time periods, allowing comparison between intern self-evaluation and parent evaluation during both the pre- and post-FCR periods (Fig 1).
Communication Skills: Intern Self-Assessments and Parent Evaluations
In the pre-FCR period, interns rated themselves significantly lower (61.3 ± 7.6) than did parents (66.7 ± 5.5, P < .05) (Fig 2A). There was no correlation between parent and intern scores (r17 = 0.05, P = .84) (Fig 2B).
In the post-FCR period, parent and intern CAT scores converged: mean-level scores did not differ (t17 < 1.0, P = .39). Parent and intern CAT scores did not differ largely because of an increase in intern self-assessment scores (Fig 3A). Furthermore, the parent-intern correlation for CAT scores was significantly positive (r18 = 0.73, P < .001) (Fig 3B). The change in parent-intern correlation is significant from pre- to post-FCR (Z = 2.21, P < .05), suggesting increased agreement between parents and residents in appraisals of interns’ skills.
In the post-FCR period, nurse CAT scores (69.6 ± 1.7) did not differ from intern self-assessment scores (65.3 ± 1.3, P = .09) or parent appraisals of interns’ communication skills (66.6 ± 2.0, P = .37). Faculty CAT scores (65.4 ± 1.4) did not differ from either intern self-assessment scores (65.3 ± 1.3, P = .97) or parent CAT scores (66.6 ± 2.0, P = .59). Neither nurse nor faculty CAT scores were correlated with scores from parents and interns. Nurse and faculty scores also did not correlate with one another. All Pearson r coefficients were below 0.20 and were not statistically significant.
Communication Skills: Comparison Between 2 Time Periods
Data on the 15 interns who had self-assessments from both time periods indicated an increase in intern CAT scores from pre-FCR (61.9 ± 2.0) to post-FCR period (64.9 ± 1.5), but that difference did not achieve statistical significance (P = .15). The correlation in intern CAT scores between these 2 time periods was positive, but not statistically significant (r15 = 0.50, P = .06). Parent appraisals of interns’ communication skills did not differ between the 2 periods (pre-FCR 65.9 ± 1.4 vs post-FCR 66.0 ± 2.2, P = .84) and were not correlated (r15 = −0.43, P = .11).
Results from this study indicate that, after the introduction of FCRs, nurses and faculty were more likely to directly observe behaviors featured on the CAT. We also observed that resident self-evaluations were aligned and significantly correlated with parent evaluations after the introduction of FCR. Finally, we found that the internal consistency of the CAT was high in all 4 evaluator groups throughout the study. Although the bedside teaching is known as an effective tool to model, observe, assess, and evaluate competencies in trainees,10,15,27,28 there are limited experimental data describing the utility of FCR for evaluation of resident communication skills. To our knowledge, this is 1 of the first studies in which the CAT was used as a multisource evaluation tool.
In an outpatient setting, bedside presentations (in the examination room) were seen as an efficient way of accomplishing several important educational goals, including direct resident observation, teaching, and provision of feedback.19 In the inpatient setting, Brinkman and colleagues5 in the study on evaluation of resident communication skills and professionalism during rounds described that attending physicians self-reported their inability to observe resident behaviors of interest, particularly in the areas of history taking, physical examination, and interpersonal skills. Similarly, faculty members in our study were previously unable to observe behaviors featured on the CAT during traditional rounds. The introduction of FCR appears to have provided additional opportunities for nurses and faculty to directly observe behaviors featured on the CAT.
On the first evaluation, before introduction of FCR, interns rated themselves lower than did parents during the pre-FCR period. This difference disappeared after FCRs were introduced, largely because of increases in intern CAT scores. Similarly to other reports, parents rated their physician on the high end of the scale throughout the study.5,21 Pre-FCR intern and parent scores were not correlated; however, after the initiation of FCR, there was a strong, positive correlation between intern self-assessment and parent ratings. The change in parent-intern correlation was significant from pre- to post-FCR period, suggesting increased agreement between parent and intern appraisals of intern communication skills. Although these findings could be the result of increased experience on the part of the interns regardless of the FCR introduction, they are also consistent with previous reports indicating that assessments between active participants in communication tend to be highly correlated, reflecting the shared perspectives of the participants.10,12 In our model of FCR, residents and parent have the most active roles in communicating, whereas an attending physician assumes the role of an educational coach.29 Although, we did not assess intern’s perceptions of FCR, it has been recently reported that medical students experienced additional “accountability" and “responsibility” when presenting during FCR.28 Therefore, it is possible that interns in our study felt they needed to prepare better when presenting at the bedside, leading to an increase in self-assessment scores. Improving physicians’ abilities to make accurate appraisals of themselves is extremely important, because self-assessment is an essential element of lifelong learning.
We also found a lack of correlation between intern and parent scores and both nurse and faculty scores during FCR. These results may indicate that nurses and attending physicians have different expectations than parents and interns in regard to competence in communication. It is also possible, as pointed out by other authors,5,23 that the concepts rated have different meaning to different raters. These null findings underscore the difficulties inherent in trying to quantify assessments that reflect qualitative judgments.23 They also raise an important question as to whose evaluation of communication and interpersonal skills should be considered the gold standard. Lack of correlation among raters leaves open the question of what are the appropriate standards for assessing the validity of ratings from different judges.23
We suggest that there is a need to define uniform performance standards and to train raters to apply them. Standardization and faculty development of FCR are essential to avoid variability in the trainee’s and patient’s experiences.28 Ottolini et al29 have successfully incorporated the Observed Structured Teaching Exercises into a FCR faculty development program to enhance faculty teaching during FCR. This approach can be tailored to specifically address assessment of communication and interpersonal skills during FCR.
Although there was an increase in intern’s communication skills after FCR introduction, this difference was not statistically significant. The lack of significance is likely attributable to inadequate statistical power arising from a small sample size. It is likely that an intervention such as the introduction of FCR did not by itself improve communication and interpersonal skills.
In this study, to evaluate resident communication and interpersonal skills, we used the CAT for all 4 groups of evaluators. Internal consistency among all evaluator groups was high during both time periods. The CAT appears to be reliable multisource instrument. It can be used by interns for self-assessment, and by the nurses and faculty to assess how well interns communicate with the parents during FCR.
This study has several limitations. First, this was a single-center study with a small sample size and included only interns, therefore limiting generalizability. Second, the changes in intern scores from pre-FCR to post-FCR are confounded by maturation that naturally occurs during the academic year. Similarly to what has been reported in previous studies, we found CAT scores to be clustered at the positive end of the scale.5,21 It appears that respondents were unwilling to give strongly negative evaluations of interns. Parents provided evaluations during a hospitalization which may have biased them to give higher scores because they were unwilling to comment candidly while under medical care.30 This response bias led to skewed scores. It is also possible that faculty may have worked with interns before, and might have been aware of their communications skills, thereby introducing additional scoring bias. Other limitations arise from measurement choices. This study did not include peer raters, although this is a crucial group of evaluators. In addition, raters did not have a formal training.
Our study suggests that FCR is useful for evaluation of residents’ competence in communication. This model of bedside rounds adds more opportunities for direct observation, and allows for multisource evaluation of trainee’s communication and interpersonal skills. The CAT can be used reliably as a multisource assessment instrument in the inpatient setting. In addition, it appears that bedside interactions between the medical team members and the family during FCR offer a climate that improves resident’s ability to self-assess.
The authors thank the members of the Division of Quality and Medical Informatics for their contributions to the shape and substance of this article, which have been substantially improved because of their thoughtful critiques.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This research was supported by a Clinical Scholar Award for Patient Safety from the Department of Pediatrics, New York-Presbyterian Hospital.
- Accreditation Council for Graduate Medical Education
- Communication Assessment Tool
- family-centered rounds
- 2.↵ACGME Outcome Project. Available at: http//www.acgme.org/Outcome/. Accessed October 2006.
- Snadden D
- Brinkman WB,
- Geraghty SR,
- Lanphear BP,
- et al
- Muething SE,
- Kotagal UR,
- Schoettker PJ,
- Gonzalez del Rey J,
- DeWitt TG
- Rosen P,
- Stenger E,
- Bochkoris M,
- Hannon MJ,
- Kwoh CK
- 13.↵Committee on Hospital Care. American Academy of Pediatrics. Family-centered care and the pediatrician’s role. Pediatrics. 2003;112(3 pt 1):691–697.
- Mittal VS,
- Sigrest T,
- Ottolini MC,
- et al
- Sectish TC,
- Zalneraitis EL,
- Carraccio C,
- Behrman RE
- 24.↵. Family Centered Care. Available at: www.cincinnatichildrens.org/about/fcc/rounds/lit-review.htm.
- Jones T,
- Sasser W
- Black NMP,
- Kelly MN,
- Black E,
- Sessums CD,
- DiPietro MK,
- Novak MA
- Copyright © 2012 by the American Academy of Pediatrics