OBJECTIVES: The Association of American Medical Colleges published a list of entrustable professional activities (EPAs) that graduating medical students should be able to perform on day 1 of residency without direct supervision. We sought to explore the perceptions of residents and pediatric hospitalists about the level of supervision new interns need in conducting these EPAs.
METHODS: An electronic survey was sent to pediatric hospitalists who supervise interns in a large pediatric residency program in which they were asked to rate the amount of supervision they perceive new interns need when performing 11 EPAs. Another survey was sent to residents in which they were asked how much supervision they needed at the beginning of their intern year when performing the same EPAs. Hospitalist and resident responses were compared.
RESULTS: The majority of hospitalists thought new interns could perform only 5 of the 11 EPAs without direct supervision. For 5 of the EPAs, residents’ perceptions of their own abilities to perform the EPA independently as interns were significantly greater than the hospitalists’ perceptions of interns’ abilities. For example, 91% of residents thought they were able to recommend and interpret common diagnostic and screening tests when they were interns without direct supervision, compared with only 30% of attending physicians (P < .001).
CONCLUSIONS: Hospitalists and residents in a pediatric residency training program reported that new interns were not consistently able to perform the Association of American Medical Colleges’ core EPAs without supervision. Furthermore, residents and pediatric hospitalists perceived new interns’ abilities to perform the EPAs without direct supervision significantly differently.
Each July, trainees advance into new roles in hospitals nationwide. Given the variation in clinical instruction across medical schools1–5 and the limited pediatric-specific training in medical school, the transition from medical school to intern year poses a particular challenge for pediatric trainees and hospitalists alike. Newly minted doctors may begin residency with considerably different knowledge bases and skill sets, and pediatric hospitalists bear the responsibility of ensuring that trainees can safely provide patient care with an appropriate level of supervision when on inpatient pediatrics rotations. There are data to suggest that inadequate supervision of residents is associated with increased medical errors6,7 and enhanced supervision of trainees is associated with improved patient outcomes,8 although the role that supervision plays in the safety of pediatric care in particular has not been well established. Additionally, across specialties, it has been demonstrated in recent literature that many program directors perceive new residents to be underprepared for residency.9
Accordingly, in May 2014, the Association of American Medical Colleges (AAMC) published a list of 13 core entrustable professional activities (EPAs) that all entering residents should be expected to perform on day 1 of residency without direct supervision, regardless of specialty.10 It is unknown, however, whether new residents are currently competent to perform the AAMC EPAs without direct supervision. To date, little work has been done examining how much supervision residents and pediatric hospitalists perceive new interns to need when performing the AAMC’s recommended EPAs Additionally, it is unknown whether pediatric hospitalists and residents have different perceptions on how much supervision is needed at the beginning of the year. We therefore sought to examine how much supervision pediatric hospitalists and residents perceive new interns to need when performing the AAMC’s core EPAs in a large pediatrics residency program.
This study was performed in a large, urban pediatric residency program. Our 4-person study group (composed of pediatric hospitalists, residency program leadership, and a chief resident) designed 2 surveys with which we intended to ascertain faculty perceptions and resident self-perceptions of the level of supervision required when performing the AAMC EPAs. We chose response options based on scales of proficiency commonly used in the literature and in similar educational studies.11 We solicited feedback from 6 content experts, including experts in competency-based education, residency education, survey design, and pediatric hospital medicine. The major revision made by this group was to exclude 2 of the 13 EPAs. We did not include the following: (1) obtaining informed consent for tests and/or procedures; and (2) performing general procedures of a physician. Although these 2 activities are considered important, interns in our pediatric residency program infrequently perform these tasks at the beginning of the intern year. Therefore, the study group worried that it would be difficult for residents and attending physicians to assess the amount of supervision needed when performing these EPAs. Before formal survey administration, we conducted a small pilot with 2 chief residents and 2 pediatric hospitalists, without major revisions. Formal cognitive interviewing was not performed. Our institutional review board reviewed the protocol and determined that it qualified as exempt.
For each of the 11 EPAs, we asked the hospitalists to answer the following question: “For each of the following activities, please rate how you would describe your level of entrustment of interns on average at the beginning of the intern year.” Answer options included the following: (1) Has knowledge; (2) May act under full supervision; (3) May act under moderate supervision; (4) May act independently; or (5) May act as a supervisor and instructor.
For each of the 11 EPAs, we asked residents the following similar question: “Rate how much supervision you feel that you needed at the beginning of the intern year when performing each of the following activities.” Answer options included the following: (1) I had knowledge about this but could not do it; (2) I was able to perform under full supervision; (3) I was able to perform under moderate supervision; (4) I was able to perform independently without supervision; or (5) I was able to function as a supervisor and instructor for this.
Approximately 4 months into the 2014–2015 academic year, the voluntary electronic survey was administered via e-mail to all pediatric hospitalists who supervise pediatric interns on inpatient general pediatric services and all current residents in our pediatrics program. Hospitalists supervise interns for a variable number of weeks per year and have a variable number of years in practice; their background and experience with EPAs was also variable. One reminder e-mail was sent 1 week after the original e-mail. Incentives were not used.
We calculated the number, percentage, and 95% confidence intervals (CIs) of pediatric hospitalists and residents who selected 1 of the top 3 options (options 3, 4, or 5 out of 5) for each activity; we determined that these 3 options best corresponded with the AAMC’s expectation that interns should be able to perform the EPAs “without direct supervision.” Fisher’s exact tests were used to test for differences in the percentages between residents on the basis of their postgraduate year (PGY) as well as differences between pediatric hospitalists and residents. A P value of <.05 was considered statistically significant. The 95% CIs were calculated by using the Wilson Method. SAS software (version 9.4; SAS Institute, Inc, Cary, NC) was used for the analysis. We performed a secondary analysis in which we limited the comparison to the top 2 options (options 4 and 5 out of 5) and excluded option 3 (performance with moderate supervision), representing an intern’s ability to perform an EPA independently without supervision.
Twenty-seven out of 50 hospitalists (54%) and 65 out of 152 possible residents (43%) responded to the survey. Of the hospitalist participants, 11 (42%) attended solely on inpatient general pediatrics wards, 7 (27%) attended on an intermediate care unit, 2 (8%) attended on a short stay unit, 6 (23%) attended on both general pediatrics and short stay units, and 1 (4%) left the question unanswered. Of the resident participants, 25 (39%) were postgraduate year 1 (PGY1), 20 (31%) were postgraduate year 2 (PGY2), 19 (29%) were postgraduate year 3 (PGY3), and 1 (2%) was postgraduate year 4 (PGY4).
Pediatric hospitalists reported that the average new intern’s ability to perform the core EPAs without direct supervision was limited (Table 1). There were only 5 EPAs that the majority of hospitalists believed new interns could perform without direct supervision, correlating to 1 of the top 3 options (options 3, 4 or 5 out of 5). In contrast, there were 9 EPAs that the majority of residents believed new interns could perform without direct supervision. On secondary analysis (using our more stringent definition in which we included only options 4 and 5), there were no activities that the majority of pediatric hospitalists thought new interns could perform independently, and there were only 3 activities that the majority of residents believed that new interns could perform independently (Table 2).
Residents perceived their own abilities to perform the EPAs without direct supervision when they were new interns significantly differently than did pediatric hospitalists (Table 1). Ninety-two percent of residents perceived that they could have prioritized a differential diagnosis after a clinical encounter without direct supervision versus only 44% of hospitalists (P < .001). In addition, 91% of residents believed they could have recommended and interpreted common diagnostic and screening tests without direct supervision versus 30% of hospitalists (P < .001). Furthermore, 72% of residents perceived that they could have given or received a patient handover to transition care responsibility without direct supervision versus 44% of hospitalists (P = .02). Lastly, 46% of residents perceived that they could have recognized a patient requiring urgent or emergent care and initiated evaluation and management versus 22% of hospitalists (P = .04). In secondary analyses in which we used the more stringent definition, the pattern of discrepancies between hospitalist and resident ratings of new interns’ abilities differed somewhat (Table 2).
For all but 1 EPA, residents across PGY levels had similar self-perceptions of their abilities to perform the activities without direct supervision at the beginning of their intern year (Table 3). However, 100% of PGY1 residents thought they could have performed the EPA “recommend and interpret common diagnostic and screening tests” at the beginning of their intern year without direct supervision, as compared with 90% of PGY2 residents and 80% of PGY3 and PGY4 residents (P = .04).
We found that pediatric hospitalists and residents in a large pediatric residency training program reported that new interns were not consistently able to perform the core EPAs that the AAMC has established as a goal for all new residents without direct supervision. For 6 of the 11 EPAs that we studied, less than half of pediatric hospitalists thought new interns could perform the EPAs without direct supervision. Even residents (who consistently indicated less need for supervision) reported that they could not have performed 2 of the 11 activities without direct supervision at the beginning of their intern year. These data support the AAMC’s call for significant curriculum development and increased training of medical students.
Less than one-third of pediatric hospitalists thought new residents could perform the following 2 EPAs without full supervision: (1) recommending and interpreting common diagnostic and screening tests; and (2) recognizing a patient requiring urgent or emergent care and initiating evaluation and management. Medical errors by new interns performing these activities can lead to significant patient harm. Accordingly, it may be unsafe to expect new interns to perform these tasks without direct supervision, although fostering their ability to do so would certainly be of value.
Although we primarily examined new residents’ performance of the EPAs without direct or full supervision, we also analyzed their abilities to perform the EPAs independently, without any supervision, as a secondary analysis. We hypothesize that many new residents do, in fact, perform many of the EPAs independently, and we therefore thought it would be important to include this analysis. For each of the activities studied, most pediatric hospitalists doubted that new interns could function independently. Similarly, more than half of residents reported that they were unable to perform 8 of the 11 tasks independently at the beginning of their intern year. Therefore, we agree with the AAMC that it is critical to provide at least some supervision for all of the EPAs.
Our study also revealed that for many of the EPAs, residents and pediatric hospitalists had significantly different perceptions regarding the need for supervision of new interns. This secondary finding is in many respects not surprising and likely in part stems from the inaccuracies of intern self-assessment. There is a robust literature base to suggest that physicians inaccurately self-assess their knowledge,12 procedural skills,12,13 and clinical competence.12,14,15 Of further concern is that trainees who are the least competent are also the least able to self-assess competently.12–14 Additionally, residents may mistakenly believe that less intern supervision is needed, perhaps because they may not fully appreciate the need for supervision or because they may have an interest in achieving autonomy as soon as possible.16,17 It is critical that pediatric hospitalists are aware of the inaccuracies of trainee self-assessment. Specifically, pediatric hospitalists should rely on direct observation in place of intern self-assessment when making entrustment decisions.
An alternative explanation for the discrepancies between hospitalist and resident ratings is that pediatric hospitalists may mistakenly believe that more intensive intern supervision is needed than is in fact the case, perhaps out of an inability to adequately evaluate trainees and a desire to ensure the safest possible care in the face of uncertainty.18 It should be noted that the EPA framework was designed with the intention of having an observer assess a single trainee in a clinical context and, on the basis of those observations, make a decision about entrustment.11 In contrast, in this survey we asked pediatric hospitalists to rate how much supervision the average trainee needs. Therefore, pediatric hospitalists are forced to reflect on all of their workplace observations of interns over many years. It is likely that specific intern interactions may unduly influence their perceptions, a form of availability bias. For example, if a pediatric hospitalist were to work with a struggling new intern in the week before completing this survey, they might mistakenly report that the average intern needs more supervision than they actually do need. Pediatric hospitalists should be aware of this potential bias and make decisions about entrustment based on direct observation rather than recent experiences.
Despite its limitations, the disparity between hospitalists’ and residents’ perceptions has some important implications for the organization of care systems on inpatient units. For example, 72% of residents self-reported that they could give or receive a patient handover to transition care responsibility at the beginning of their intern year without direct supervision, as compared with only 44% of hospitalists. Researchers in a previous study have demonstrated that implementation of a supervised group handover was associated with reduced medical errors.19 Therefore, if interns do, in fact, perform handoffs independently or with only moderate supervision, then adverse patient outcomes could result. Reassuringly, however, there is agreement between residents and pediatric hospitalists regarding some of the other high-stakes activities (eg, entering and discussing orders and prescriptions).
Interestingly, there was little difference in the perceived need for supervision of new interns between residents across PGY levels. One may have hypothesized that as residents progressed in training, they may have changed their self-perceptions on their need for supervision at the beginning of their intern year; however, this occurred to only a limited degree. The stability in answers across PGY levels also suggests that PGY3 residents, who are further away from the beginning of their intern year and could theoretically be at risk for recall bias, may in fact recall how much supervision they perceived they needed at the start of internship reasonably accurately.
There are several limitations of this study. First, the response rate was modest, and there may be a nonresponse bias. Additionally, we did not specify in the surveys who would be supervising the new interns (ie, residents, pediatric hospitalists, or both), and therefore the responses were based on the interpretation of the respondents. It is possible that residents might have assumed this supervision would be by residents, whereas hospitalists assumed it would be by attending physicians. We also did not define some of the terms used in the survey (eg, “average intern” and “moderate supervision”), and therefore these terms were subject to interpretation. In addition, pediatric hospitalists were asked to rate the amount of supervision that the average intern needs at the beginning of the intern year, whereas residents were asked to rate how much supervision they themselves needed at the beginning of their intern year. It is possible that residents might either under- or overestimate how much supervision they themselves needed as compared with their peers. Additionally, this study was performed in a single large pediatric program, and therefore its results may not be generalizable across other hospitals. Lastly, shifting expectations regarding intern competence and the evolution of hospitalists’ clinical expertise could affect how ratings were provided.
With further preparation at the end of medical school or the onset of internship, we believe that it will be possible to narrow the gap between the current and desired independence of graduating medical students and new interns. We are encouraged by new fourth-year medical school boot camps and other curricula that aim to better prepare new interns.20 In addition, residency programs could consider implementing intensive training programs for incoming interns on their arrival.21 Such training programs would permit residency program leadership and pediatric hospitalists to simultaneously assess new residents’ capabilities and adjust the levels of individual supervision accordingly. Given pediatric hospitalists’ unique opportunity to assess new interns and tailor their supervision on the basis of learner competency, we should offer faculty development regarding intern assessment and feedback. By better preparing our new interns to enter residency and by improving pediatric hospitalists’ abilities to tailor supervision, we can better align intern and attending physician expectations. It is essential that residents and hospitalist physicians openly discuss these issues so that interns can be supervised appropriately and patients can receive optimal care.
The authors thank all of the residents and faculty who worked closely with us and participated in this study.
FINANCIAL DISCLOSURE: Dr Landrigan has served as a paid consultant to Virgin Pulse to help develop a sleep and health program. He is supported in part by the Children’s Hospital Association for his work as an executive council member of the Pediatric Research in Inpatient Settings network. In addition, Dr Landrigan has received monetary awards, honoraria, and travel reimbursement from multiple academic and professional organizations for teaching and consulting on sleep deprivation, physician performance, handoffs, and safety, and he has served as an expert witness in cases regarding patient safety and sleep deprivation. Dr Sectish is a consultant to the I-PASS Institute and has equity in the company. He has also received monetary awards, honoraria, and travel reimbursement from multiple academic organizations for his work on patient safety research. We do not believe these activities to be conflicts of interest for this study. Drs Winn, Marcus, and Williams have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Supported by a Boston Children’s Hospital residency funding grant titled the Fred Lovejoy Resident Research and Education Award.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Barzansky B,
- Etzel SI
- Barzansky B,
- Etzel SI
- Barzansky B,
- Etzel SI
- Barzansky B,
- Etzel SI
- ↵Association of American Medical Colleges. Core entrustable professional activities for entering residency. 2013. https://www.mededportal.org/icollaborative/resource/887. Accessed January 15, 2016.
- Evans AW,
- Aghabeigi B,
- Leeson R,
- O’Sullivan C,
- Eliahoo J
- Biondi EA,
- Varade WS,
- Garfunkel LC,
- et al
- Meyerson SL,
- Teitelbaum EN,
- George BC,
- Schuller MC,
- DaRosa DA,
- Fryer JP
- Starmer AJ,
- Landrigan CP
- Englander R,
- Flynn T,
- Call S,
- et al
- Winn AS,
- Marcus CH,
- Williams K,
- et al
- Copyright © 2018 by the American Academy of Pediatrics