Objective: The goal of this study was to determine which measurable factors of resident training experience contribute to improvement of in-training examination (ITE) and certifying examination (CE) scores.
Methods: This is a descriptive retrospective study analyzing data from July 2003 through June 2006 at a large academic pediatric training program. Pediatric categorical residents beginning residency in July 2003 were included. Regression analyses were used to determine if the number of admissions performed, core lectures attended, acute care topics heard, grand rounds attended, continuity clinic patients encountered, or procedures performed correlated with improvement of ITE scores. These factors were then analyzed in relation to CE scores.
Results: Seventeen residents were included in this study. The number of general pediatric admissions was the only factor found to correlate with an increase in ITE score (P = .04). Scores for the ITE at pediatric levels 1 and 3 were predictive of CE scores. No other factors measured were found to influence CE scores.
Conclusions: Although all experiences of pediatric residents likely contribute to professional competence, some experiences may have more effect on ITE and CE scores. In this study, only general pediatric admissions correlated significantly with an improvement in ITE scores from year 1 to year 3. Further study is needed to identify which elements of the residency experience contribute most to CE success. This would be helpful in optimizing residency program structure and curriculum within the limitations of duty hour regulations.
Residency is a multifactorial entity that serves to provide comprehensive training for physicians in a particular specialty area. Each specialty has a certifying body to conduct an examination that provides objective information on the medical knowledge a physician-in-training has obtained. For pediatrics, the American Board of Pediatrics (ABP) certifying examination (CE) is the objective measure of clinical knowledge acquired through pediatric residency. During pediatric residency, medical knowledge is assessed annually through the in-training examination (ITE). ABP data show that pediatric level (PL)-3 ITE scores correlate with CE pass rates.1 In addition, ITE scores have been shown to predict success on other specialty certifying examinations.2–5 ITE scores, therefore, may serve as a marker for individual deficiencies in medical knowledge and, possibly, deficiencies within a residency curriculum.
Few published data currently exist to describe factors associated with improvement in ITE scores within pediatric residency programs. Studies of surgical and emergency medicine programs have primarily looked at the effect of organized conferences. Findings have been variable. Organized conferences accounted for a slight increase in ITE scores in 4 studies from surgical residency programs.4,6–8 Conversely, data from 3 other studies (1 emergency medicine, 2 surgery residency programs) failed to demonstrate an increase in ITE scores attributed to teaching conferences.9–11 An emergency medicine residency program reviewed scores before the institution of an organized conference and compared those results with scores achieved after and found that improvements on the ITE were not accounted for by the initiation of the conference.10 Procedural skills, in a surgery residency, in particular, did not significantly impact ITE scores.12 Other studies have looked at targeted academic interventions on residents who have performed poorly on the ITE. These interventions have not consistently demonstrated improvement on examinations.12,14 None of the studies attempted to quantify clinical encounters along with didactic experiences and relate these to examination scores.
With recent residency duty hour restrictions, pediatric residency curriculums have been restructured with few data to guide them. In the absence of data, program directors are left to base these decisions on service needs and overall residency requirements. Knowledge of what experiences positively influence examination scores may assist with development of targeted interventions or even help guide the restructuring of residency programs. In this study, we sought to determine the extent to which measurable clinical and didactic factors of a large academic pediatric residency training program contribute to improvement in ITE scores from year 1 to year 3 of residency and subsequent performance on the CE.
This is a retrospective descriptive study taking place at a large academic pediatric residency program from July 1, 2003 through June 30, 2006. Study participants included the categorical pediatric residents beginning residency training in July 2003. Pediatric residents were excluded if they were in noncategorical or combined residency programs because of the concern for confounding factors associated with exposure to different residency curriculums. Because of the ever-changing residency requirements and program structure, we examined 1 class to control for variations in required rotations, call structure, and duty hours that may contribute to differences among classes.
Institutional review board waiver was obtained from the participating center. Study residents were not aware of this study at any point during training.
The PL-1 and PL-3 ITE scores were compared to determine the change in ITE score as 1 dependent variable. CE score was recorded for the residents who took the CE in 2006 as another dependent variable.
Independent variables examined inclzded admissions performed, clinic encounters recorded, procedures recorded, and didactic sessions attended.
These variables were further broken down as described below.
The number of inpatient floor admissions performed by each resident at both the intern and the supervisory resident level were counted by using admission data collected and archived by the chief residents. This information is collected for all floor admissions 7 days a week, including weekends and holidays, as part of routine program procedure. The admissions were subdivided into general and subspecialty admissions with the exception that PICU and NICU data are not tracked.
Continuity clinic patient and procedure logs are maintained by each resident throughout their training. As part of routine program policy, continuity patient and procedure logs were reviewed by the program director with the residents; the residents signed off on their logs as an accurate reflection of continuity patients seen and procedures done at the end of residency.
In this pediatric residency training program, there are 3 main didactic components: Core Curriculum, Grand Rounds, and Acute Care Symposium. Signing in was the responsibility of the resident. As part of routine program procedure, attendance records were reviewed by the program director with the residents biannually; residents signed off on their attendance as an accurate reflection of experiences at the end of residency.
Core Curriculum is an 18-month series of lectures that covers topics from both general and subspecialty pediatrics with a curriculum that is geared toward the general pediatrician. Grand Rounds is conducted once weekly. An unknown case is presented by one of the residents involved in the patient’s care, and the audience asks questions to figure out the diagnosis. A wrap-up discussion is then provided by an expert on that topic. The Acute Care Symposium is a month-long lecture series conducted each July on common medical emergencies in various specialties.
The PL-1 and PL-3 ITE scores were compared to determine the change from PL1 to PL-3. Descriptive statistics were used to analyze characteristics of all variables including examination scores. Primary measures recorded were median and range.
Linear regression and correlation analyses were done to examine the relationship between the dependent variables (change in ITE score and CE score) and each independent variable to determine if any of the variables described above significantly contributed to the change in ITE score. Results of coefficient of determination (R2), correlation coefficient (r), P values, and line fit plots were recorded and examined. Analyses were run on subgroups of admissions (general pediatrics, subspecialty, intern and senior resident) to see if any particular type of admission had effect. Similarly, analysis was done on the subgroups of didactics (Core Curriculum, Acute Care, Grand Rounds) in addition to total didactics numbers. For the 16 of these residents who took the CE in 2006, the same independent variables were analyzed to determine if any significantly predicted CE performance. In addition, PL-1 ITE, PL-3 ITE, and change in ITE score were analyzed to see if there was significant correlation with CE score.
Seventeen categorical pediatric residents were included in this study. Two residents scored zero on the PL-1 ITE. One of these zero-score residents did not complete the CE. Analyses were done with and without the zero-score residents’ data and were comparable (Table 1).
The median and range values for admissions, clinic encounters, procedures performed, and didactic sessions attended can be seen in Table 2. The median for admissions performed was 615 with the majority being general pediatric admissions. Clinic encounters and procedures logged varied. Medians and ranges were similar with and without the zero-score residents’ data.
When linear regression analysis was performed by using each independent variable, only the number of general pediatrics admissions was found to significantly correlate with an increase in ITE score with all residents included. This correlation remained even when excluding the zero-score residents. No other variable significantly correlated with change in ITE score (Table 3).
The data for the 16 residents from this cohort who took the CE for the first time after completing residency training was used. For the CE, no variable was found to correlate significantly with higher scores (Table 3).
For the 16 residents who took the CE, the PL-1 ITE score correlated significantly with CE score with all scores included (r 0.731, P = .001) and with the zero score excluded (r 0.65, P = .009). The PL-3 ITE score correlated with the CE score when the zero-score resident data were included (r 0.582, P = .02), but this relationship did not remain significant when that data were excluded (r 0.46, P = .08). The change in ITE score from year 1 to year 3 did not correlate with CE score.
In this study, we sought to identify factors that contributed to improvement in pediatric ITE scores and subsequent scores on the ABP CE. This study is unique, because we examined both clinical and didactic experiences. None of the factors contributed to higher CE scores, and only general pediatric admissions correlated significantly with an improvement in ITE scores from year 1 to year 3. It was surprising that the total number of admissions did not correlate with score improvement. One would think more is always better, but at least these data showed that only more of general pediatric admissions correlated with score improvement. This may reflect the patients with increasingly complex, chronic disease on subspecialty inpatient services that, in general, are not the subject of the general pediatric CE. Thus, other than potentially maximizing general pediatric inpatient experiences, it is still unclear how pediatric residency programs can direct experiences to improve performance on standardized CEs.
The scores on the ITE taken in year 1 and 3 were predictive of higher CE scores. Althouse and McGuiness1 published data showing that PL-3 ITE scores are predictive of CE scores but that the predictive power of the ITE in the first training year is minimal. Our finding that a higher PL-1 ITE score was predictive of a higher CE score suggests that there are factors intrinsic to the test taker that contribute to successful examination performance. These findings suggest that residents who test well will continue to do so regardless of the educational activities of residency that we were able to assess.
Innate factors that could contribute to examination success include critical thinking, comprehension, attention/memory, and functioning under stress. Critical thinking is not formally taught in residency but is important to both examination and medical practice success. One would like to think we are teaching our trainees how to think, but often we probably teach them just to follow attending direction or protocols. The correlation of general pediatrics admissions with higher change in ITE examination scores may reflect the opportunity for trainees to have more autonomy and independent thought on those patients. Focused observation and feedback on resident documentation and presentation could help hone comprehension and attention/memory skills. Training on functioning in stressful situations also could be beneficial. Interventions to improve such skills may help residents be more successful on their future CE rather than solely focusing on medical knowledge acquisition. Studies could be done looking at the effect of such interventions on CE scores, but one must carefully consider if achieving better test scores is the goal of residency.
Key limitations of this study were small sample size, recall bias for self-reported measures, and the inability to measure other factors like self-study. We chose to look at 1 class for several reasons despite knowing it would limit our sample size. Classes before and after had significant changes to duty hours and program structure during their residency. The use of >1 class would potentially confound results, because each class had significantly different exposures. The other key limitation was possible measurement bias due to reliance on self-reported data for procedures, clinic encounters, and didactic attendance records (based on residents remembering to sign in). A wide range of numbers was reported for both of these, possibly reflecting differences in resident compliance with recording. It is possible that our numbers could be an over- or underestimate of actual resident activities in these areas. Furthermore we did not assess other factors that might influence examination scores such as self-study, reading, and practice test question use.
Our hope is that this study spurs further research in this area because larger, more comprehensive studies are needed. Having national standardization of measures of resident competence would make future research in residency education easier and likely more significant. Hopefully, the Accreditation Council for Graduate Medical Education shift to the Next Accreditation System with milestones will provide such tools. Additional work needs to be done to provide residency directors data on which to make program structure decisions. Currently, service coverage often trumps educational value for our residents.
There is still no clear answer for how residency programs can optimize their structure and curriculum within the limitations of duty hour regulations to specifically improve resident performance on the CE. It seems that maximizing general pediatric patient encounters may be helpful. As Accreditation Council for Graduate Medical Education regulations and residency structures evolve, future studies in this area are clearly warranted.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: No external funding.
Dr Chase performed the data collection and data analysis, critically reviewed and revised the manuscript, and approved the final manuscript as submitted; Dr Highbaugh-Battle performed data analysis, drafted the initial manuscript, and approved the final manuscript as submitted; and Dr Buchter conceptualized and designed the study, coordinated and supervised data collection and analysis, critically reviewed the manuscript, and approved the final manuscript as submitted.
- American Board of Pediatrics
- certifying examination
- in-training examination
- pediatric level
- Copyright © 2012 by the American Academy of Pediatrics