This article is intended for those interested in learning more about how advances in selection science can benefit training programs and applicants. This may be of particular interest to residency and fellowship program directors. The objectives are to:
- Describe the various ways in which our current selection system is costing both applicants and programs
- Provide resources and examples of how advances in selection science can benefit training programs and applicants
- Discuss ideas for next steps in advancing selection efforts in surgery
Less than a month ago, more than 1,200 medical students found out they successfully matched into a general surgery residency program.1 Aside from basking in the excitement of their future careers, they are also likely feeling relieved they have successfully navigated the gauntlet of The Match
Residency programs are likely similarly relieved they filled all of their positions, as they will have invested anywhere from $61,000-$129,000 in time and resources reviewing applications, conducting interviews, and creating rank lists.6
Unfortunately, if past data is any indication of future data, this “match” won’t last long. More than one in five interns entering a surgery residency program won’t make it to the graduation finish line.7 The reasons for this high attrition rate are likely multifaceted, but program directors report that those who leave join a different surgery program (25 percent), join another surgical discipline (52 percent), or leave medicine entirely (21 percent).8 Additionally, at least a third of those remaining will undergo at least one remediation attempt, most often to enhance “soft skills” such as professionalism or interpersonal skills.9–10
These data are concerning for a number of reasons, but they particularly highlight the opportunities for improvement that exist within our current residency selection process. Fortunately, there is an entire field of personnel psychology dedicated to improving the process of selection. What follows are three areas in which selection science may be able to offer insight and value for the surgical education community:
1. Efficiency
As most of the current selection costs are incurred for the on-site visit, a logical starting point for both parties would be to reduce the number of interviews without compromising selection quality. Selection science offers a number of tools to help in this regard; one of the most powerful is a situational judgment test (SJT). SJTs are written exams that present the candidate with real-life situations they will likely encounter during residency.11–13 Programs are able to judge how applicant responses match those they would expect from a resident in their program, and applicants are able to get a realistic preview of what would be expected from them during training.
Another powerful selection tool is the structured interview. Unlike the typical “get acquainted” interview conducted for most candidates,14–15 a structured interview focuses on asking questions that determine if a candidate has the specific attributes the program has identified as important for success. For these, it is critical that faculty are trained on conducting structured interviews and using the rating forms.
A surgical fellowship recently incorporated online SJTs and structured interviews into their application process. They were able to reduce the number of on-site interviews by more than 50 percent while still matching top-ranked candidates.16 Exit surveys indicated that applicants believed the application and interview process was more organized in comparison to other programs and gave them better insight into their fit with the program.17 Surveyed faculty felt empowered to make better selection decisions based on objective data.
2. Objectivity & Equity
The foundation of any assessment is fairness and freedom from subjectivity or bias.18 Unfortunately, the current process by which residency candidates are assessed—standardized test scores, letters of recommendation, and personal statements14—may not meet this criteria. As mentioned earlier, the USMLE is most often the primary screening tool for further consideration. However, the USMLE is prone to issues of fairness, given research that cognitive-based examinations result in substantial racial differences in test performance.19 The exam is also not predictive of residency performance beyond future test scores,20 so it likely eliminates candidates from consideration that might overall perform well in the program. In addition, numerous studies have shown sex bias in letters of recommendation,21–23 with women receiving less standout adjectives and ability words, compared to men even when objective criteria indicates no differences in qualifications. Personal statements can demonstrate bias as well, as they may handicap applicants who lack access to individuals in positions of power or prestige to help with letter review or coaching.24 Finally, unstructured interviews, which are often given the most weight in final ranking decisions,25 are subject to a number of biases26 and prone to inappropriate questions related to candidate demographics.27
Evidence-based selection focuses on objectivity and equity. Tools, such as personality tests, SJTs, and structured interviews, can offer important information about a candidate’s fit within a program, without incorporating high levels of bias like more traditional screening tools. In fact, preliminary research within surgery suggests that these tools can actually increase diversity in training programs16 if used in the application screening process.
3. Reduced Attrition
Although the reasons for resident attrition are likely multi-faceted, one of the primary reasons for dropout may be lack of fit within the program. By designing selection systems that are based on past resident successes (and failures), residency programs can narrow in on competencies that have historically been indicators of resident fit (or misfit). For example, Kelz et al.28 worked with an organizational management expert to review historical reasons for resident dismissals, which included deficiencies in stress management, organizational skills, future aspirations, and prioritization abilities. From this, they re-structured their application and interview process to focus on screening for these specific competencies. The authors stipulated that this novel selection strategy reduced attrition from an overall five-year rate of 27 percent to 3.2 percent.
Surgery is a profession that cares about people. In addition to caring for patients, it is critical that we utilize systems that allow us to care about the newest members of our profession—our applicants—and those on the education team who are tasked with managing the high volume of applications received. By adopting principles from selection science, we have the opportunity to incorporate fair, efficient, and evidence-based screening tools that allow candidates to learn more about our training program while we learn more about them.
